Welcome back for another episode of Pivotal Conversations. In this episode, guest host Derek Harris interviews Karun Chanuri and Ramesh Krishnaram from T-Mobile about applying chaos engineering to Pivotal Cloud Foundry. So what you're going to hear is really about what makes chaos engineering different in a Pivotal Cloud Foundry environment. What are the differences between running it for Pivotal Application Service versus the Pivotal Container Service? Now, you're also going to hear about their thought process on how they found and came up with the chaos engineering tool set that they ultimately built and have contributed in the open source. You're also going to hear about how they onboarded teams and prepared them for running their first game day earlier this year. Now, some interesting things also came out around why microservices have really challenged and caused chaos engineering to evolve from the early days of pulling cables directly out of systems in a data center. Now, if you want to hear more about what's going on at T-Mobile and how their chaos engineering practice has evolved on Pivotal Cloud Foundry, be sure to register to attend SpringOne Platform, where colleagues of Karun and Ramesh are actually presenting the part two of this story. So you can register now and get a $200 discount by using code S1P200 underscore D Draywitz. That's my first initial and last name just for you. So listen up and enjoy. Okay. And we're here with our latest and greatest, hopefully greatest podcast from CF Summit. I'm here with Ramesh Krishnarum and Karun Shinuri from T-Mobile. Welcome to Pivotal Conversations, guys. Thank you, Derek.
2:00Thank you for having us. Yeah. So before we get started, you know, and diving into the meat of chaos engineering, can you guys just give a quick, and I mean quick, like job titles, backgrounds, so people have a sense of where you're coming from? Sure. Hi. This is Ramesh Krishnarum. I'm the senior manager for the platform engineering team at T-Mobile. I've been with T-Mobile over a year now, around 17 months, approximately. And in a nutshell, my team manages the container strategy for T-Mobile, so we're essentially the drivers for the next generation platform like PCF and PKS. Hi. My name is Karun Shinuri. I'm a senior engineer platform engineering team, so I take care of security architecture and the container strategy and also work in the team where we try to onboard customers onto Cloud Foundry and onto the Kubernetes. All right, cool. I'll assume most people listening have some background they're familiar with with PCF and they're familiar with Kubernetes, so maybe we don't need to dive into that too deeply, but I think maybe what people are less familiar with is they have chaos engineering, right? So can you kind of, I mean, you guys have presented to a lot of people, and go check it out on YouTube, some of your various talks, but can you have, I mean, the quick background, at least, chaos engineering, how you think about it? Yeah, I can get started and Karun, feel free to jump in on points that I missed out, right? So the principles of chaos engineering really advocates building more resilient services so you can avoid catastrophic customer-impacting events in production. It was a concept that was fairly initiated in the early days when Netflix was just getting into the business of microservices, and they were the pioneers in this area who wrote a book on chaos engineering. And the principles of chaos engineering is really about building more secure, resilient microservices and introducing failures on purpose so you can break your services to see what kind of failures happen. And the reason you want to do that is when engineers write services, you make assumptions.
4:01Your design has assumptions. Some of the assumptions you make are your network being infinite, your bandwidth is unlimited, your network is secure. So these are some of the classic assumptions that you make, and if some of these assumptions don't go validated by running actual tests, then you're prone to failures. And when these failures happen, the impact is on your customers. You think about how Netflix has evolved from being one of those startup companies to a much more big, resilient company these days, right? They actually run chaos attacks in production, live these days, during daytime. They don't tell you when they run it. They take down a complete AC. And that's their adoption rate for chaos engineering internally with their services. So we're just trying to start off small, thinking about the platform that we are delivering to our customers at T-Mobile, essentially with Cloud Foundry and Kubernetes. We're labeled the most critical platform, so there's a lot of due diligence behind it. There's a lot of responsibility on our shoulders, right? We don't necessarily want to run attacks on production yet. The first step in this journey for us is to embrace the principles of chaos, understand how it can benefit my team and my own customers. For example, for my own team, we're delivering all these massive capabilities on an infrastructure behind the scenes. And there are things that could go wrong from an infrastructure standpoint for compute, network, and storage. And then there's applications being deployed within the infrastructure where we deliver capabilities and things could go wrong there. So the last few months, our journey has been mostly around writing this toolkit and getting that culture embraced internally with the toolkit to see how we can build more resilient services. Maybe you can add more about the toolkit itself. So it's all about conducting the experiments on your production environment and see if your systems are resilient to the common failures or not. So we started identifying chaos engineering into two different ways, like one on the infrastructure side of things and one on the application side of things. So what happens, like Ramesh said, if infrastructure goes down, how would your applications deployed on the infrastructure react, right? And what happens if you want to perform a very targeted attack, like application-level chaos engineering,
6:05how would your application behave when there are some unfavorable experiments or some kind of attacks are done on the application and its dependencies? So that's what, in a nutshell, is called chaos engineering. All right. And where are you on this journey? It seems like you'll be doing it for several months at least. Yeah. I'd say we, well, when I was actually exposed to the principles of chaos engineering, even before my role at T-Mobile, same thing applies to Corona. We can talk about the history, but that's history. Let's talk about the current scenario. So when we started delivering on these container strategies, clearly there were people on the team, like Corona and I, thinking about, okay, we've got to start thinking about chaotic attacks on our own infrastructure and also building more resiliency for our applications. Because oftentimes, in the very early days of last year, I was looking at some of the anti-patterns that exist within our own infrastructure as well as the applications delivered. So we started looking at some commercial evaluation of tools out there for chaos engineering because we necessarily didn't want to reinvent the wheel. So we looked at a couple of open source tools and also a commercial offering and did the pros and cons analysis. I'll let him speak for the pros and cons analysis. But the output of that was, okay, we needed a toolkit that's not only just doing infrastructure attacks, but also application level attacks, right? Because like I said, it's a combination of both. I want to run attacks on infrastructure, compute network and storage, and then there's also application level attacks, right? And that's the challenge. There was not a toolkit that would do both. And really, the driving force for us was to embrace open source culture, which is we wanted to deliver something to open source T-Mobile. And there was a bunch of active projects there because we wanted to give it back to the community and make it a more community-driven contribution there, right? So it's just a journey we started off on. And the capabilities of the tool started slowly getting built out. Over the last few months, we've made progression in the toolkit, in fact, really good progression in terms of getting the toolkit publicized and even advertised with a game day with one of our own customers internally, which we just ran a game day a couple of weeks ago. And a game day is a practice where we get an application team, run targeted attacks
8:07with their permission in a non-prod environment, and be doctors giving them prescription on what they could do better to build more resilient apps on the platform. Yeah, I was going to ask about that. It seems like, to my, if I think about it, I can think of one commercial offering that's targeted at Chaos Engineering, and then all this stuff Netflixed. Yeah. So at least it's basically where my mind goes. Yeah, so there's different toolkits available in the market, and even in open source world, starting with Chaos Monkey, Turbulence, and different toolkits, it's not a one-size-fits-all model, unfortunately, right? So, because every infrastructure and every application ecosystem is unique. Right. And for us, it was more giving back to the Cloud Foundry community and the Kubernetes community, because that's where a lot of the open source contributions are. So, one important point here is, from the architecture standpoint, when I started technically leading this project, one thing that struck my mind is, these are the systems like Cloud Foundry in general and PKS, they are deployed via Bosch. So there are not many open source solutions or the commercial offerings who can perform Chaos Engineering for the Bosch-deployed environments. So, we soon realized that, you know, okay, the commercial offerings that we have in the market, they are not going to work. And we started talking to the vendors, the commercial, you know, the company name, company, and then they came up with the toolkit, but still, it's not good enough for meeting our requirements. So, we started building our own toolkit around Cloud Foundry. And now, the toolkit that we have is, can understand how Bosch operates, and it can perform the chaos attacks specific to the Bosch and clusters managed by the Bosch. All right, you know, that makes a lot of sense, yeah. I think it's probably easy to target one. It's easy to target maybe AWS, or it's easy to target Kubernetes, but it has to target Kubernetes and Bosch and specific applications. Yeah, so for AWS and other places, you have Chaos Monkey and Chaos Kong. There are so many toolkits there, but again, they are not Bosch-specific.
10:10So, what we have today is Bosch-specific chaos. All right. Is chaos engineering, like, is this something, you know, you hear it discussed as companies move into microservice and move into distributed systems. Is there a reason that this wasn't a practice previously in a model? No, it's just, if you think about it, engineering discipline has been evolving over the several years, right? There's industrial engineering, there is, you know, aerospace engineering, different kinds of engineering. This is the newest discipline of engineering, which is chaos engineering. And it's the culmination of all the lessons learned by some of the top engineering companies, like Facebook, Netflix, and some of the great people that are coming together to say, okay, failure is an inevitable part of your life, so you need to focus on how can you avoid failures in a more confined environment. And that's how chaos engineering was really born. So, it's a fairly new concept, but it is, like, a fairly new concept that is getting a lot of traction and adoption. So, that's my answer. So, the concept was there even previously when people, a couple of years back, when we were in the waterfall model, where everything was monolithic, the only way to test these failures was to plug out the cable from the data center and see how it works. That's a good point, yeah. Now, the complexity is around microservices. These are small little microservices which are designed and built to do one thing and one thing to perfection, right? So, there's a lot of communication, and hundreds and thousands of these microservices talking to each other. It's not possible to plug out something, right? So, that's where you need a sophisticated toolkit, and that's where the science around chaos engineering has evolved. So, it's all about simulation. Like, he said, great point. I did not brush on that, but really, it's about simulating these attacks, right, as opposed to just randomly plugging out a cable. That's not what we're doing here. We're simulating network latencies. We're simulating network failures, and the toolkit is really helping you there, so you don't have to physically sit there and yank cables out. So, what is good enough? I mean, given that failure is inherent in a distributed system, what does good enough look like from a chaos engineering perspective?
12:19When you mean good enough, you know, what kind of attacks are good enough? Or what does resiliency look like? I mean, how do you say, okay, we feel pretty comfortable with this? Yeah, so, that's a good question, actually. You know, this is a common recommendation we make, right? You know, no matter how 12-factored your application is. So, there are some common best practices when you're designing your application in the microservice world. So, there are, like, 12-factor practices that we've been brushing to our customers quite often on this day. But, in spite of following all these best practices, there are some places, like, you know, soft spots in your application, which are prone to a single point of failure. So, identifying those soft spots, which you may or consciously or unconsciously might have designed the system and deployed it in the production. So, what happens if there's a single point of failure? So, that's where the chaos engineering is coming in. And the general recommendation from our side is make sure whatever the app you have, it is auto-scaled and it has, you know, cloned, it is cloned into multiple foundations. That way, you are highly available. So, security talks about three important aspects, confidentiality, integrity, and availability. Availability is a key role for any application to be available. You're 24 by 7, 365 days. That's something that, you know, if you have to make that available, then your application should be 12-factored and it should be running in multiple foundations. And it should be, it should have multiple instances to take the load. So, that's an important aspect that we're trying to target. Another thing is on what is good enough, it's a difficult question to answer, but it's good enough if you get to a point where you have apps running in production and you can successfully take those apps down and not impact the customer. Ultimately, it matters to the customers. If you work backwards from your customers and you say, okay, I took my apps down and I'm still able to place orders on T-Mobile's website. That should be the experience that we should be striving for. Can I do device upgrades by taking down dependent services in production, right? Because the principles of chaos engineering advocates that you run these tests in production, but it's a graduation phase.
14:22You start off simple in non-production environments or staging environments and then graduate to running it in product, right? So, the driving force should be, can you get to a point where you can run these continuous experiments in production and not impact your customer? And that is the desired state. Yeah, and the user experience is so very important here. Like, you know, okay, if there is something down, the user doesn't need to know, okay, something was really down. Yeah, I mean, I guess maybe I was getting to an 80-20 rule where you feel happy. Like, you're happy if you feel like you've accounted for. Yeah. Because you saw, like, just recently, right, like, all of Facebook's. Yeah. I mean, Facebook is a master of this, and yet it was out for, what, 14 hours? Yeah. The 80-20 rule would be, at least in my mind, and let Karun speak for him, like, for us, right now, the focus is on the outages that we had over the last, let's say, one year. Looking at the historic outages, doing an outage analysis study to see where the common points of failure are, which are very customer-impacting, and how can chaos engineering help there? If it's a repeatable pattern, right, that means that it's, and if it's addressable by principles of chaos, by running chaotic attacks, we should target on those things and not just look at code fixes. Code fixes is one thing, but really running those simulations outside of production is important. Right. So, T-Mobile, we started looking at more, taking more proactive approach than reactive approach. So, for being proactive, we have to have some historical data. So, when we dig into the incident reports for the last couple of years, we could see, we could identify some common patterns. Some of them could be easily avoided with resiliency tests that could have been done, like, you know, earlier and be better prepared, right? So, being proactive is something very important, you know, when it comes to chaos engineering. All right. I know you talked about, right, their infrastructure-level concerns or application-level concerns. Is there a nightmare scenario? Is there that thing where, like, God forbid? It's every day, because, you know, as much as we are very proud about our, us being one of the biggest footprint with PCF and now towards footprint with PKS, it's that thing that keeps you awake, right?
16:29Like, behind the scenes, there is this big responsibility. So, for me, it's everything. Simple changes that could impact our customers. Over the last few weeks, I'm not going to name the outages. We've had some outages which could have been averted, either through proactive testing or by failure testing, right? So, yeah, I can't really answer what keeps you up at night. The infrastructure and the scale at which we operate keeps me up at night. But Cloud Foundry, in general, is pretty resilient in this matter. Like, you know, the whole concept of self-healing that Cloud Foundry brings in, like, you know, it's pretty good in that aspect. When something goes down, it brings it back. But there are some other areas, like storage, that Cloud Foundry depends on. That is where the resiliency and the soft spots are in the architecture. So, we have to keep an eye on that. Interesting. Is that a matter of those parts of the infrastructure just getting up to speed? I know that's been on the work and storage, but it's still kind of, like, it's in storage, right? Yeah, so for us, there's, you know, we've grown so quickly. So, there are some single point of failures in our architecture, a lot of which we've addressed last year, a lot we're going to address this year. And as we address these single points of failure, it's not just fixing it, but also validating what we've actually deployed, right? The S3 Blob Store, for example, that's behind our infrastructure, which is responsible or a key element of all the CF push enablement, right? We've run into this space issues with that. So, we've done a lot of re-architecture there. But oftentimes, we do the re-architecture. Are we running any load testing or are we running any failure testing on that infrastructure, right? That's the piece that I want to aspire towards this year. How can I validate that this is rock solid? Those are the things that we should be focusing on next, not just re-architecture, but running actual tests to validate our assumptions. Do the challenges come more from, as you grow, is it more from just the sheer scale? Is it, like, more services equals more issues or is it just, like, larger footprints, right? It's a combination of everything.
18:31If you look at our footprint, I mentioned some numbers today, 39,000 containers, 13 plus foundations. We're proud of some of these numbers. We're also not proud of some of the numbers because at the scale at which we're growing, we grew vertically to a point where some of our foundations are a nightmare to manage. I saw a talk on that. Yeah. And you saw Brandon's talk on how big is too big, right? There are challenges when you grow vertically and then you realize that your upgrades become very, very painful. Even simple repave operations are going to take quite a bit of time, right? So, we're addressing those by right-sizing foundations this year, building out more foundations, focusing on automation being the key element. In fact, one of the security vision talks about automation being, you know, a big driving force for how you deploy changes to your environments, right? It should be a modular, consistent, and repeatable process. It's been proven that in the information security space, anything that is automated has less error rate in terms of security. All right. I mean, you mentioned earlier, you know, obviously working with the application teams and you talked about game days and stuff. So, how does that work? I mean, how do you interact with developers and application teams whose applications? Yeah, there is an onboarding process. I'll slightly pressure upon it, but then I'll let Karun talk about the actual game day. Our product manager, Shipra, was also involved in it. She's a big driving force also, right? I'll talk about the high-level pitch. So, we had the toolkit ready. We came up with a process for onboards where we reached out to some of our key customers. We announced that we have a toolkit with which we want to run game days. And we gave them a perspective of what the game days is, what the onboarding process looks like. The onboarding process is answering some questions. In our questionnaire, we evaluate them and then we decide a date on when we want to meet them for the game days. And then once the game days run, we give them a prescription report of what else they could be doing better, right? Maybe you want to zoom in a little more on this. So, the first thing is we have a platform with 3,000 applications running and belonging to multiple customers.
20:35So, when we are conducting these game days, infrastructure-level chaos attack is ruled out. We didn't want to do it because it's going to impact several other customers. So, we have to get a justification from that. So, when we are doing this game day with a particular customer, we take their application, we find their, like, you know, how their design and architecture is, and take a wild guess, like, you know, fair guess, like, you know, what exactly may go wrong. So, we draft our experiments. So, when, on the day of game day, we conducted, like, two hours session. So, we announced the first experiment that we are going to conduct. And we expect the app team sitting on the other side of the table to take a guess. They have to investigate, detect, and also remediate what happened, how to fix that particular, you know, problem. So, let's say if we are injecting a failure test, they have to identify that this is the particular failure that we have triggered. Yeah. So, 10 out of the 10 plus teams you have reached, 6 have expressed interest, and we have a waiting list of people that want to go in with us, right? So, but there is equal amount of preparatory work that application teams will have to do in terms of getting the questionnaire filled up, having that design discussions with us, and then targeting a day for game day, right? What environments are we going to run the test on? So, we've been able to successfully do that with one team. But moving into Q2, like Karun said, we want to, you know, repeat rinse this test with a few other teams. In fact, I want to actually push some failure injection tests to the infrastructure level, but that will need a little more of a buy-in and sign-off from my own team, right? Because we consider our non-prod environment such as prod, so it's not as easy as just going and launching a network latency attack on a non-prod environment, because they're going to impact several hundred application teams. Right. Yeah, that seemed to be the challenge. Is there, like, how widely, like, I mean, how many application teams does the company have, right, and is the goal to hit this? I mean, I guess it depends on how fast CF adoption and PKS adoption happens, right? Yeah, the adoption for CF has been monumental.
22:35PKS is getting there. So, I think from an adoption standpoint, that's not a challenge. The challenge really has been, there is the what about the platform and how to use the platform. The challenge has been how to use the platform, right? Because if you know how to use the platform, you can build better resilient apps. That's the challenge that we're having at T-Mobile, and I don't necessarily see adoption as a challenge. Everybody wants to use our platform, to be honest with you. So, it's just coaching them on how to use the platform, and we're considering this as an opportunity to teach them how to use the platform. Earlier on, Karun mentioned on the 12-factor principles, which is conformance to standards of building right applications, right? A lot of the times, we see that application teams have their priorities around getting their code pushed to prod sooner, which means they may have to bypass some of the constraints on conformance to standards, which is essentially signing up for tech debt later on when an outrage happens. Just doing these targeted attacks with us or, you know, with these game day sessions with us will help them mature before they run their apps in production at scale. All right. How good do you think developers can actually get in terms of designing for resiliency, right? Because, I mean, there's no perfect, obviously, right? So, is there a point where, like, a level of, I don't know, adequacy might be the word, that would be the baseline? So, that's where this 12-factor comes into picture. So, that's the reason, like, you know, when you're designing an application, go through the standards and make sure you're compliant with it. And if you're following, at least, I'm sure, like, 60 to 70% of your application is resilient by default. And Cloud Foundry will fill in the rest of the gap. But still, there could be some external dependencies with the apps running inside the Cloud Foundry. Let's say if it is talking to an external database, what happens if the external database is down? You know, how is your application going to behave? So, those are the external factors that application team has to keep an eye on and have the resiliency designed for it, both from the, in a holistic way, not just for the apps that are in Cloud Foundry, but also for the apps that it is dependent on.
24:37Yeah. I have, I think T-Mobile has really, some really sharp developers that, because we've been interacting closely with them, right? But, again, like I mentioned, the challenge is going to be achieving that balance between their application development versus building resilient apps. And achieving the challenge is going to be a tough one, right? It is a culture change. Just like adopting a new platform is a culture change, the DevOps culture change I'm talking about, this is going to be yet another culture change, which is going to be something that is going to be done slowly but surely and steadily. And where I was going with that, I'm trying to miss the point now where I'm going with that. So, actually, it's not a one-stop solution for all the applications. So, every application is different. Every application that's deployed inside the PCF and PKS has their own needs. So, yes, we leave it to the developers to decide, and all the architects have a key role to play in that. Do you think, do you feel like you have or should have control over, or maybe I'm trying to get, maybe there's a framework that gets in place around these external pieces, right? I mean, the database, right? Yeah. These other pieces, because, obviously, those are going to become a major failure point, too. Yeah, you're talking about, like, a governance board that reviews architecture? Or, yeah, or just as, you know, a standard, like, short of some, like, the presumption you're deploying with this database. Sure, sure. Yeah, most of the teams have principal engineers and a member of technical staff embedded in their application team, so, who make a lot of really good design-driven decisions. We're being closely collaborating with some of the top, you know, minds at T-Mobile to form that governance body or center of excellence. In fact, there's some driving conversations already to create that center of excellence, which, you know, teams that are going through the microservice adoption will go through the center of excellence for some design patterns and analysis. Yeah, so, one thing we, as a platform engineering team, we do not do any more. We're focusing on how the apps are getting deployed. We can only educate, but there is design reviews that happens at the security end. So, but, again, it depends on every application that has to go to the production has to go through the process of design reviews within the team.
26:41So, that's why I'm curious to say, like, you know, you talked earlier about Netflix and some of the companies that pioneered chaos engineering. How much do you think you can actually, and you touched on this, how much can you actually learn from those companies when you, you know, when you go read, like, a Netflix blog post and they go in there, go see a talk? You know, or how much of it really is, like, it's bespoke to T-Mobile just because the platform and the application, they're so different. Yeah. Again, for me, it's a culture change, right? It has to be done slowly. If I go backwards in time to March or April of 2018, this was just getting started and we had no idea what we wanted to do. So, right, and then we got out on the right track. Today, we are here talking in front of you. We just did our first keynote. We had our deep divest today. So, we're getting the momentum or getting the advertisement done. We have customers lined up with us to do game day. So, I think the adoption is going to be steady as more people adopt PKS and PCF. Chaos engineering is a known concept to all development teams. What's not known is the framework that we have created. So, we're advertising the framework now. And I think the adoption for, you know, fast forwarding one year from now is going to be much bigger. And I can guarantee you that there will be at least 15 different game day sessions we might have done by then, right? And this is probably a larger conversation at that point. Think about Netflix. Netflix started this in 2011 or early 2010. And they were in the sweet spot because from the get-go, they started looking at AWS as their infrastructure. AWS was just getting started. So, it was a lot of partnership. And their history is also important here because people that went from Amazon to Netflix to create their Chaos Kong or Chaos Monkey framework, or people that worked at Amazon that knew failure testing and took the practices to Netflix. So, they were trying to get their culture going from the get-go that this is the right way to do it, right? For us, we're a little backwards here because we have a lot of legacy applications that are going through this transformation phase. So, I'm fairly confident that if we get to a point where we run these game days with a lot more teams, you know, we'll get to a point where we'll have much more better resilient apps in production.
28:45Right. I think that's the thing that gets overlooked sometimes is you read and you see, well, Netflix, like, so obviously, it could be easy for us, except, right, there's legacy, there's culture, there's all these things, there's institutional knowledge. Yeah. There's all these things that, like, most companies just don't have. And then there's leadership buy-in also. That's extremely important, right? Because if you don't have that buy-in, then you'll go nowhere with these things, unfortunately. So, the important thing is we know what our applications are. We have 80% of our applications built in Spring Boot framework, so Java and Spring Boot. So, whatever the toolkit that we are coming up with, it is generic enough, but it is more focused on the Spring applications as well. All right. And then I just want to, you know, kind of in closing here, I want to ask, is there, I mean, we talk, you know, I guess I tend to think of PCF in broad terms. And obviously, you guys are running on PAS and PKS. Is there, do you have to look at things differently if you're talking about applications or services running on PAS versus PKS? Are there, I would imagine, architecturally, right, there are just different gotchas out there. I'll take a stab, and then I'll let Karun know. PCF versus PKS, right? Yeah. So, before any customer is getting onboarded, we try to look at their application and try to divert them into the right platform that the application can be hosted on. So, if, for example, if the application that's been developed is a Spring Boot and which has a dependency with the config server, service registry, then, you know, obviously go with PCF. PKS is not going to help them out in that aspect. But in cases where customers are coming up with their own container, like Docker containers, then, obviously, PKS is their ideal choice. I think the other follow-up question you had was, is chaos engineering different for apps built on PCF and PKS? Not really, because under the hood, it's all Bosch-controlled and deployed. For us, it's one framework that should work seamlessly for apps built on PCF and PKS. Because our infrastructure is the same for us. So, again, the chaos toolkit that we are building is more specific to Bosch-deployed environment. Okay, yeah. So, since PCF and PKS are both backed with Bosch, the Turbulence++ is a toolkit that we have.
30:48We have an engineer, Matthew, who has been extensively working on this. Conor Matthew is part of my team. So, we have plans of trying to deploy this Turbulence++ and Monarch into the PKS environment in the near future. Okay, and then just to close, guys, I mean, you've touched on some of this already, so maybe don't feel free not to repeat yourself too heavily. But, I mean, you mentioned, like, okay, you got, like, a handful of one team or a handful of teams, you know, bought in and ready. I mean, what is, what are the end goals, I guess, for the chaos engineering aspect of your job? For me, it's, if these fields, like, the team with which we ran game days, this is a team that does coverage map analysis, and it's a front-facing app. We ran attacks, several kinds of attacks, simulated attacks in their non-prod environment. If they feel comfortable that their app is mature, resilient, and if they are willing to bet that let's target these attacks in prod, that's the big winner for me, right? But it's really them taking our prescriptions, making sure that they feel comfortable that they can run these attacks in production. In fact, this team is running their workloads in PCF, PKS, and also in the public cloud. So, it's a great team to run such attacks because we could take, we could run our attacks in one and around one and see how it impacts the customers. That's the desired state. So, oftentimes, like, you know, if teams are a little shy coming to the game days, we also wanted to come up with a concept of providing it as a service, wherein developers or the development teams can use that as a service, chaos engineering as a service, and perform their attacks as and when they want. Yeah, so part of that, what Karun's talking about is, you know, your continuous integration and deployment pipeline has automated tests as part of your CICD as code goes from one stage of the pipeline to another, right? So, what he and I are envisioning is why not just have failure testing as part of the CICD? So, it happens in a lower end environment. So, as code matures and goes to Prague, this chaos engineering as a service will target these failure tests, and good quality code gets pushed to production.
32:50Oh, that's clever. So, we're just taking out that extra. Yes. That extra stuff. So, rather than having them coordinate with us if they're shy, they're more than welcome to use our framework, integrate it with their test framework, and just integrate it with their pipelines, and just naturally have the code go through this sort of maturity. All right, cool. Yeah, that makes a lot of sense. Anything else? Anything we didn't touch on? No, this is great. A lot of great questions. Thank you for having us. All right. Thanks for being on, guys. Yeah, thank you.