1 00:00:00,000 --> 00:00:02,740 Welcome back for another episode of Pivotal Conversations. 2 00:00:03,260 --> 00:00:11,400 In this episode, guest host Derek Harris interviews Karun Chanuri and Ramesh Krishnaram from T-Mobile 3 00:00:11,400 --> 00:00:16,580 about applying chaos engineering to Pivotal Cloud Foundry. 4 00:00:16,920 --> 00:00:22,500 So what you're going to hear is really about what makes chaos engineering different in a 5 00:00:22,500 --> 00:00:24,220 Pivotal Cloud Foundry environment. 6 00:00:24,220 --> 00:00:28,980 What are the differences between running it for Pivotal Application Service versus the 7 00:00:28,980 --> 00:00:30,300 Pivotal Container Service? 8 00:00:30,820 --> 00:00:35,660 Now, you're also going to hear about their thought process on how they found and came 9 00:00:35,660 --> 00:00:40,800 up with the chaos engineering tool set that they ultimately built and have contributed 10 00:00:40,800 --> 00:00:42,000 in the open source. 11 00:00:42,520 --> 00:00:46,620 You're also going to hear about how they onboarded teams and prepared them for running their first 12 00:00:46,620 --> 00:00:48,380 game day earlier this year. 13 00:00:49,060 --> 00:00:55,780 Now, some interesting things also came out around why microservices have really challenged 14 00:00:55,780 --> 00:01:02,580 and caused chaos engineering to evolve from the early days of pulling cables directly out 15 00:01:02,580 --> 00:01:04,880 of systems in a data center. 16 00:01:05,580 --> 00:01:11,260 Now, if you want to hear more about what's going on at T-Mobile and how their chaos engineering 17 00:01:11,260 --> 00:01:18,060 practice has evolved on Pivotal Cloud Foundry, be sure to register to attend SpringOne Platform, 18 00:01:18,060 --> 00:01:24,860 where colleagues of Karun and Ramesh are actually presenting the part two of this story. 19 00:01:24,860 --> 00:01:34,100 So you can register now and get a $200 discount by using code S1P200 underscore D Draywitz. 20 00:01:34,320 --> 00:01:37,060 That's my first initial and last name just for you. 21 00:01:37,300 --> 00:01:39,020 So listen up and enjoy. 22 00:01:43,120 --> 00:01:43,840 Okay. 23 00:01:44,420 --> 00:01:51,340 And we're here with our latest and greatest, hopefully greatest podcast from CF Summit. 24 00:01:51,560 --> 00:01:56,360 I'm here with Ramesh Krishnarum and Karun Shinuri from T-Mobile. 25 00:01:56,860 --> 00:01:59,140 Welcome to Pivotal Conversations, guys. 26 00:01:59,780 --> 00:02:00,220 Thank you, Derek. 27 00:02:00,460 --> 00:02:01,320 Thank you for having us. 28 00:02:01,640 --> 00:02:01,800 Yeah. 29 00:02:01,840 --> 00:02:05,440 So before we get started, you know, and diving into the meat of chaos engineering, can you 30 00:02:05,440 --> 00:02:09,580 guys just give a quick, and I mean quick, like job titles, backgrounds, so people have 31 00:02:09,580 --> 00:02:10,460 a sense of where you're coming from? 32 00:02:10,720 --> 00:02:10,920 Sure. 33 00:02:11,400 --> 00:02:11,580 Hi. 34 00:02:11,640 --> 00:02:12,720 This is Ramesh Krishnarum. 35 00:02:12,720 --> 00:02:16,640 I'm the senior manager for the platform engineering team at T-Mobile. 36 00:02:17,500 --> 00:02:22,600 I've been with T-Mobile over a year now, around 17 months, approximately. 37 00:02:23,320 --> 00:02:28,500 And in a nutshell, my team manages the container strategy for T-Mobile, so we're essentially 38 00:02:28,500 --> 00:02:33,040 the drivers for the next generation platform like PCF and PKS. 39 00:02:33,820 --> 00:02:34,200 Hi. 40 00:02:34,460 --> 00:02:35,680 My name is Karun Shinuri. 41 00:02:35,900 --> 00:02:41,620 I'm a senior engineer platform engineering team, so I take care of security architecture 42 00:02:41,620 --> 00:02:48,220 and the container strategy and also work in the team where we try to onboard customers 43 00:02:48,220 --> 00:02:50,280 onto Cloud Foundry and onto the Kubernetes. 44 00:02:51,040 --> 00:02:51,500 All right, cool. 45 00:02:51,860 --> 00:02:56,980 I'll assume most people listening have some background they're familiar with with PCF 46 00:02:56,980 --> 00:03:01,860 and they're familiar with Kubernetes, so maybe we don't need to dive into that too deeply, 47 00:03:01,860 --> 00:03:08,500 but I think maybe what people are less familiar with is they have chaos engineering, right? 48 00:03:08,760 --> 00:03:12,580 So can you kind of, I mean, you guys have presented to a lot of people, and go check it out on 49 00:03:12,580 --> 00:03:17,020 YouTube, some of your various talks, but can you have, I mean, the quick background, 50 00:03:17,140 --> 00:03:19,820 at least, chaos engineering, how you think about it? 51 00:03:20,520 --> 00:03:24,700 Yeah, I can get started and Karun, feel free to jump in on points that I missed out, right? 52 00:03:24,760 --> 00:03:29,240 So the principles of chaos engineering really advocates building more resilient services so 53 00:03:29,240 --> 00:03:32,820 you can avoid catastrophic customer-impacting events in production. 54 00:03:33,520 --> 00:03:38,700 It was a concept that was fairly initiated in the early days when Netflix was just getting 55 00:03:38,700 --> 00:03:44,860 into the business of microservices, and they were the pioneers in this area who wrote a book 56 00:03:44,860 --> 00:03:46,520 on chaos engineering. 57 00:03:46,520 --> 00:03:51,900 And the principles of chaos engineering is really about building more secure, resilient microservices 58 00:03:51,900 --> 00:03:56,760 and introducing failures on purpose so you can break your services to see what kind of failures happen. 59 00:03:57,060 --> 00:04:01,180 And the reason you want to do that is when engineers write services, you make assumptions. 60 00:04:01,740 --> 00:04:02,880 Your design has assumptions. 61 00:04:03,140 --> 00:04:09,480 Some of the assumptions you make are your network being infinite, your bandwidth is unlimited, your network is secure. 62 00:04:09,880 --> 00:04:15,400 So these are some of the classic assumptions that you make, and if some of these assumptions don't go validated by running actual tests, 63 00:04:15,820 --> 00:04:16,900 then you're prone to failures. 64 00:04:17,300 --> 00:04:19,660 And when these failures happen, the impact is on your customers. 65 00:04:20,220 --> 00:04:27,500 You think about how Netflix has evolved from being one of those startup companies to a much more big, resilient company these days, right? 66 00:04:27,800 --> 00:04:33,420 They actually run chaos attacks in production, live these days, during daytime. 67 00:04:33,700 --> 00:04:34,860 They don't tell you when they run it. 68 00:04:34,900 --> 00:04:36,060 They take down a complete AC. 69 00:04:36,760 --> 00:04:42,020 And that's their adoption rate for chaos engineering internally with their services. 70 00:04:42,680 --> 00:04:48,560 So we're just trying to start off small, thinking about the platform that we are delivering to our customers at T-Mobile, 71 00:04:48,780 --> 00:04:50,300 essentially with Cloud Foundry and Kubernetes. 72 00:04:51,140 --> 00:04:55,280 We're labeled the most critical platform, so there's a lot of due diligence behind it. 73 00:04:55,660 --> 00:04:57,880 There's a lot of responsibility on our shoulders, right? 74 00:04:57,940 --> 00:05:00,280 We don't necessarily want to run attacks on production yet. 75 00:05:00,860 --> 00:05:04,240 The first step in this journey for us is to embrace the principles of chaos, 76 00:05:04,440 --> 00:05:07,140 understand how it can benefit my team and my own customers. 77 00:05:07,940 --> 00:05:13,900 For example, for my own team, we're delivering all these massive capabilities on an infrastructure behind the scenes. 78 00:05:14,400 --> 00:05:17,960 And there are things that could go wrong from an infrastructure standpoint for compute, network, and storage. 79 00:05:18,740 --> 00:05:24,740 And then there's applications being deployed within the infrastructure where we deliver capabilities and things could go wrong there. 80 00:05:25,140 --> 00:05:28,260 So the last few months, our journey has been mostly around writing this toolkit 81 00:05:28,260 --> 00:05:33,740 and getting that culture embraced internally with the toolkit to see how we can build more resilient services. 82 00:05:34,480 --> 00:05:36,300 Maybe you can add more about the toolkit itself. 83 00:05:36,300 --> 00:05:39,840 So it's all about conducting the experiments on your production environment 84 00:05:39,840 --> 00:05:43,060 and see if your systems are resilient to the common failures or not. 85 00:05:43,180 --> 00:05:48,320 So we started identifying chaos engineering into two different ways, 86 00:05:48,460 --> 00:05:52,200 like one on the infrastructure side of things and one on the application side of things. 87 00:05:52,660 --> 00:05:55,920 So what happens, like Ramesh said, if infrastructure goes down, 88 00:05:56,740 --> 00:05:59,920 how would your applications deployed on the infrastructure react, right? 89 00:06:00,300 --> 00:06:05,160 And what happens if you want to perform a very targeted attack, like application-level chaos engineering, 90 00:06:05,840 --> 00:06:10,860 how would your application behave when there are some unfavorable experiments 91 00:06:10,860 --> 00:06:15,160 or some kind of attacks are done on the application and its dependencies? 92 00:06:15,660 --> 00:06:18,160 So that's what, in a nutshell, is called chaos engineering. 93 00:06:18,840 --> 00:06:19,380 All right. 94 00:06:19,640 --> 00:06:24,180 And where are you on this journey? 95 00:06:24,240 --> 00:06:26,500 It seems like you'll be doing it for several months at least. 96 00:06:26,640 --> 00:06:26,780 Yeah. 97 00:06:27,240 --> 00:06:31,800 I'd say we, well, when I was actually exposed to the principles of chaos engineering, 98 00:06:31,920 --> 00:06:35,420 even before my role at T-Mobile, same thing applies to Corona. 99 00:06:35,520 --> 00:06:37,620 We can talk about the history, but that's history. 100 00:06:37,720 --> 00:06:39,340 Let's talk about the current scenario. 101 00:06:39,760 --> 00:06:41,900 So when we started delivering on these container strategies, 102 00:06:42,080 --> 00:06:46,120 clearly there were people on the team, like Corona and I, thinking about, 103 00:06:46,220 --> 00:06:49,300 okay, we've got to start thinking about chaotic attacks on our own infrastructure 104 00:06:49,300 --> 00:06:51,280 and also building more resiliency for our applications. 105 00:06:51,280 --> 00:06:54,760 Because oftentimes, in the very early days of last year, 106 00:06:54,880 --> 00:06:58,640 I was looking at some of the anti-patterns that exist within our own infrastructure 107 00:06:58,640 --> 00:07:00,020 as well as the applications delivered. 108 00:07:00,560 --> 00:07:05,480 So we started looking at some commercial evaluation of tools out there for chaos engineering 109 00:07:05,480 --> 00:07:08,120 because we necessarily didn't want to reinvent the wheel. 110 00:07:08,240 --> 00:07:11,380 So we looked at a couple of open source tools and also a commercial offering 111 00:07:11,380 --> 00:07:13,860 and did the pros and cons analysis. 112 00:07:14,080 --> 00:07:15,600 I'll let him speak for the pros and cons analysis. 113 00:07:15,780 --> 00:07:19,600 But the output of that was, okay, we needed a toolkit that's not only just doing 114 00:07:19,600 --> 00:07:22,160 infrastructure attacks, but also application level attacks, right? 115 00:07:22,180 --> 00:07:23,940 Because like I said, it's a combination of both. 116 00:07:24,320 --> 00:07:27,480 I want to run attacks on infrastructure, compute network and storage, 117 00:07:27,480 --> 00:07:29,640 and then there's also application level attacks, right? 118 00:07:29,680 --> 00:07:30,440 And that's the challenge. 119 00:07:30,540 --> 00:07:32,180 There was not a toolkit that would do both. 120 00:07:32,620 --> 00:07:36,220 And really, the driving force for us was to embrace open source culture, 121 00:07:36,340 --> 00:07:38,880 which is we wanted to deliver something to open source T-Mobile. 122 00:07:39,100 --> 00:07:42,260 And there was a bunch of active projects there because we wanted to give it back to the community 123 00:07:42,260 --> 00:07:44,640 and make it a more community-driven contribution there, right? 124 00:07:44,680 --> 00:07:46,460 So it's just a journey we started off on. 125 00:07:46,960 --> 00:07:50,560 And the capabilities of the tool started slowly getting built out. 126 00:07:52,020 --> 00:07:56,100 Over the last few months, we've made progression in the toolkit, in fact, really good progression 127 00:07:56,100 --> 00:07:59,660 in terms of getting the toolkit publicized and even advertised with a game day 128 00:07:59,660 --> 00:08:03,460 with one of our own customers internally, which we just ran a game day a couple of weeks ago. 129 00:08:03,680 --> 00:08:07,820 And a game day is a practice where we get an application team, run targeted attacks 130 00:08:07,820 --> 00:08:09,740 with their permission in a non-prod environment, 131 00:08:10,420 --> 00:08:13,060 and be doctors giving them prescription on what they could do better 132 00:08:13,060 --> 00:08:14,640 to build more resilient apps on the platform. 133 00:08:14,840 --> 00:08:15,540 Yeah, I was going to ask about that. 134 00:08:15,540 --> 00:08:20,760 It seems like, to my, if I think about it, I can think of one commercial offering 135 00:08:20,760 --> 00:08:25,580 that's targeted at Chaos Engineering, and then all this stuff Netflixed. 136 00:08:25,800 --> 00:08:25,920 Yeah. 137 00:08:26,040 --> 00:08:27,720 So at least it's basically where my mind goes. 138 00:08:28,820 --> 00:08:32,040 Yeah, so there's different toolkits available in the market, 139 00:08:32,140 --> 00:08:35,080 and even in open source world, starting with Chaos Monkey, Turbulence, 140 00:08:35,160 --> 00:08:38,960 and different toolkits, it's not a one-size-fits-all model, unfortunately, right? 141 00:08:39,040 --> 00:08:42,120 So, because every infrastructure and every application ecosystem is unique. 142 00:08:42,120 --> 00:08:42,260 Right. 143 00:08:42,660 --> 00:08:45,440 And for us, it was more giving back to the Cloud Foundry community 144 00:08:45,440 --> 00:08:49,640 and the Kubernetes community, because that's where a lot of the open source contributions are. 145 00:08:49,760 --> 00:08:53,560 So, one important point here is, from the architecture standpoint, 146 00:08:54,020 --> 00:08:56,480 when I started technically leading this project, 147 00:08:56,960 --> 00:08:58,840 one thing that struck my mind is, 148 00:08:59,960 --> 00:09:04,240 these are the systems like Cloud Foundry in general and PKS, 149 00:09:04,880 --> 00:09:06,920 they are deployed via Bosch. 150 00:09:07,300 --> 00:09:11,200 So there are not many open source solutions or the commercial offerings 151 00:09:11,200 --> 00:09:15,540 who can perform Chaos Engineering for the Bosch-deployed environments. 152 00:09:16,300 --> 00:09:19,060 So, we soon realized that, you know, okay, 153 00:09:19,200 --> 00:09:21,320 the commercial offerings that we have in the market, 154 00:09:21,580 --> 00:09:22,800 they are not going to work. 155 00:09:23,360 --> 00:09:26,180 And we started talking to the vendors, the commercial, you know, 156 00:09:26,420 --> 00:09:30,760 the company name, company, and then they came up with the toolkit, 157 00:09:30,760 --> 00:09:34,140 but still, it's not good enough for meeting our requirements. 158 00:09:34,260 --> 00:09:37,800 So, we started building our own toolkit around Cloud Foundry. 159 00:09:38,240 --> 00:09:42,940 And now, the toolkit that we have is, can understand how Bosch operates, 160 00:09:43,160 --> 00:09:49,020 and it can perform the chaos attacks specific to the Bosch and clusters managed by the Bosch. 161 00:09:49,340 --> 00:09:51,120 All right, you know, that makes a lot of sense, yeah. 162 00:09:51,240 --> 00:09:54,100 I think it's probably easy to target one. 163 00:09:54,660 --> 00:09:57,880 It's easy to target maybe AWS, or it's easy to target Kubernetes, 164 00:09:58,100 --> 00:10:01,580 but it has to target Kubernetes and Bosch and specific applications. 165 00:10:01,580 --> 00:10:05,340 Yeah, so for AWS and other places, you have Chaos Monkey and Chaos Kong. 166 00:10:05,520 --> 00:10:10,380 There are so many toolkits there, but again, they are not Bosch-specific. 167 00:10:10,880 --> 00:10:13,240 So, what we have today is Bosch-specific chaos. 168 00:10:13,240 --> 00:10:13,480 All right. 169 00:10:13,760 --> 00:10:21,000 Is chaos engineering, like, is this something, you know, you hear it discussed as companies move into microservice 170 00:10:21,000 --> 00:10:22,420 and move into distributed systems. 171 00:10:22,800 --> 00:10:28,380 Is there a reason that this wasn't a practice previously in a model? 172 00:10:28,540 --> 00:10:33,180 No, it's just, if you think about it, engineering discipline has been evolving over the several years, right? 173 00:10:33,240 --> 00:10:37,660 There's industrial engineering, there is, you know, aerospace engineering, different kinds of engineering. 174 00:10:38,060 --> 00:10:40,660 This is the newest discipline of engineering, which is chaos engineering. 175 00:10:41,200 --> 00:10:46,020 And it's the culmination of all the lessons learned by some of the top engineering companies, 176 00:10:46,020 --> 00:10:51,240 like Facebook, Netflix, and some of the great people that are coming together to say, 177 00:10:51,320 --> 00:10:53,920 okay, failure is an inevitable part of your life, 178 00:10:53,980 --> 00:10:57,740 so you need to focus on how can you avoid failures in a more confined environment. 179 00:10:58,260 --> 00:10:59,960 And that's how chaos engineering was really born. 180 00:11:00,040 --> 00:11:06,580 So, it's a fairly new concept, but it is, like, a fairly new concept that is getting a lot of traction and adoption. 181 00:11:07,780 --> 00:11:09,080 So, that's my answer. 182 00:11:09,120 --> 00:11:14,540 So, the concept was there even previously when people, a couple of years back, 183 00:11:14,580 --> 00:11:17,580 when we were in the waterfall model, where everything was monolithic, 184 00:11:18,240 --> 00:11:24,060 the only way to test these failures was to plug out the cable from the data center and see how it works. 185 00:11:24,420 --> 00:11:25,280 That's a good point, yeah. 186 00:11:25,760 --> 00:11:28,260 Now, the complexity is around microservices. 187 00:11:28,860 --> 00:11:36,120 These are small little microservices which are designed and built to do one thing and one thing to perfection, right? 188 00:11:36,240 --> 00:11:40,820 So, there's a lot of communication, and hundreds and thousands of these microservices talking to each other. 189 00:11:41,240 --> 00:11:43,040 It's not possible to plug out something, right? 190 00:11:43,380 --> 00:11:48,940 So, that's where you need a sophisticated toolkit, and that's where the science around chaos engineering has evolved. 191 00:11:48,940 --> 00:11:50,200 So, it's all about simulation. 192 00:11:50,440 --> 00:11:51,300 Like, he said, great point. 193 00:11:51,480 --> 00:11:57,840 I did not brush on that, but really, it's about simulating these attacks, right, as opposed to just randomly plugging out a cable. 194 00:11:58,600 --> 00:11:59,760 That's not what we're doing here. 195 00:11:59,800 --> 00:12:01,060 We're simulating network latencies. 196 00:12:01,280 --> 00:12:07,180 We're simulating network failures, and the toolkit is really helping you there, so you don't have to physically sit there and yank cables out. 197 00:12:07,580 --> 00:12:09,460 So, what is good enough? 198 00:12:09,980 --> 00:12:18,700 I mean, given that failure is inherent in a distributed system, what does good enough look like from a chaos engineering perspective? 199 00:12:19,800 --> 00:12:22,500 When you mean good enough, you know, what kind of attacks are good enough? 200 00:12:22,740 --> 00:12:24,760 Or what does resiliency look like? 201 00:12:24,840 --> 00:12:27,900 I mean, how do you say, okay, we feel pretty comfortable with this? 202 00:12:28,380 --> 00:12:30,500 Yeah, so, that's a good question, actually. 203 00:12:30,640 --> 00:12:33,280 You know, this is a common recommendation we make, right? 204 00:12:33,360 --> 00:12:36,360 You know, no matter how 12-factored your application is. 205 00:12:36,700 --> 00:12:40,700 So, there are some common best practices when you're designing your application in the microservice world. 206 00:12:41,060 --> 00:12:46,240 So, there are, like, 12-factor practices that we've been brushing to our customers quite often on this day. 207 00:12:46,240 --> 00:12:55,740 But, in spite of following all these best practices, there are some places, like, you know, soft spots in your application, which are prone to a single point of failure. 208 00:12:56,280 --> 00:13:03,280 So, identifying those soft spots, which you may or consciously or unconsciously might have designed the system and deployed it in the production. 209 00:13:03,600 --> 00:13:06,060 So, what happens if there's a single point of failure? 210 00:13:06,060 --> 00:13:09,180 So, that's where the chaos engineering is coming in. 211 00:13:09,620 --> 00:13:22,280 And the general recommendation from our side is make sure whatever the app you have, it is auto-scaled and it has, you know, cloned, it is cloned into multiple foundations. 212 00:13:22,640 --> 00:13:24,380 That way, you are highly available. 213 00:13:24,880 --> 00:13:29,580 So, security talks about three important aspects, confidentiality, integrity, and availability. 214 00:13:30,340 --> 00:13:34,080 Availability is a key role for any application to be available. 215 00:13:34,200 --> 00:13:35,960 You're 24 by 7, 365 days. 216 00:13:36,440 --> 00:13:43,860 That's something that, you know, if you have to make that available, then your application should be 12-factored and it should be running in multiple foundations. 217 00:13:44,200 --> 00:13:47,020 And it should be, it should have multiple instances to take the load. 218 00:13:47,420 --> 00:13:49,580 So, that's an important aspect that we're trying to target. 219 00:13:49,680 --> 00:14:01,660 Another thing is on what is good enough, it's a difficult question to answer, but it's good enough if you get to a point where you have apps running in production and you can successfully take those apps down and not impact the customer. 220 00:14:01,840 --> 00:14:04,040 Ultimately, it matters to the customers. 221 00:14:04,040 --> 00:14:10,020 If you work backwards from your customers and you say, okay, I took my apps down and I'm still able to place orders on T-Mobile's website. 222 00:14:10,420 --> 00:14:13,020 That should be the experience that we should be striving for. 223 00:14:13,120 --> 00:14:17,180 Can I do device upgrades by taking down dependent services in production, right? 224 00:14:17,220 --> 00:14:21,920 Because the principles of chaos engineering advocates that you run these tests in production, but it's a graduation phase. 225 00:14:22,300 --> 00:14:27,320 You start off simple in non-production environments or staging environments and then graduate to running it in product, right? 226 00:14:27,380 --> 00:14:34,100 So, the driving force should be, can you get to a point where you can run these continuous experiments in production and not impact your customer? 227 00:14:34,260 --> 00:14:35,160 And that is the desired state. 228 00:14:35,160 --> 00:14:37,620 Yeah, and the user experience is so very important here. 229 00:14:37,840 --> 00:14:42,300 Like, you know, okay, if there is something down, the user doesn't need to know, okay, something was really down. 230 00:14:43,200 --> 00:14:48,560 Yeah, I mean, I guess maybe I was getting to an 80-20 rule where you feel happy. 231 00:14:48,680 --> 00:14:51,660 Like, you're happy if you feel like you've accounted for. 232 00:14:51,900 --> 00:14:52,000 Yeah. 233 00:14:52,120 --> 00:14:54,900 Because you saw, like, just recently, right, like, all of Facebook's. 234 00:14:55,200 --> 00:14:55,340 Yeah. 235 00:14:55,440 --> 00:14:58,440 I mean, Facebook is a master of this, and yet it was out for, what, 14 hours? 236 00:14:58,680 --> 00:14:58,820 Yeah. 237 00:14:59,440 --> 00:15:09,020 The 80-20 rule would be, at least in my mind, and let Karun speak for him, like, for us, right now, the focus is on the outages that we had over the last, let's say, one year. 238 00:15:09,300 --> 00:15:17,540 Looking at the historic outages, doing an outage analysis study to see where the common points of failure are, which are very customer-impacting, and how can chaos engineering help there? 239 00:15:17,540 --> 00:15:27,080 If it's a repeatable pattern, right, that means that it's, and if it's addressable by principles of chaos, by running chaotic attacks, we should target on those things and not just look at code fixes. 240 00:15:27,460 --> 00:15:31,820 Code fixes is one thing, but really running those simulations outside of production is important. 241 00:15:32,200 --> 00:15:32,420 Right. 242 00:15:32,520 --> 00:15:37,640 So, T-Mobile, we started looking at more, taking more proactive approach than reactive approach. 243 00:15:37,800 --> 00:15:41,820 So, for being proactive, we have to have some historical data. 244 00:15:42,180 --> 00:15:48,800 So, when we dig into the incident reports for the last couple of years, we could see, we could identify some common patterns. 245 00:15:48,940 --> 00:15:57,880 Some of them could be easily avoided with resiliency tests that could have been done, like, you know, earlier and be better prepared, right? 246 00:15:57,880 --> 00:16:03,640 So, being proactive is something very important, you know, when it comes to chaos engineering. 247 00:16:04,020 --> 00:16:04,320 All right. 248 00:16:04,540 --> 00:16:10,360 I know you talked about, right, their infrastructure-level concerns or application-level concerns. 249 00:16:10,460 --> 00:16:12,120 Is there a nightmare scenario? 250 00:16:12,420 --> 00:16:15,900 Is there that thing where, like, God forbid? 251 00:16:16,680 --> 00:16:29,200 It's every day, because, you know, as much as we are very proud about our, us being one of the biggest footprint with PCF and now towards footprint with PKS, it's that thing that keeps you awake, right? 252 00:16:29,280 --> 00:16:31,420 Like, behind the scenes, there is this big responsibility. 253 00:16:31,760 --> 00:16:33,720 So, for me, it's everything. 254 00:16:34,060 --> 00:16:36,200 Simple changes that could impact our customers. 255 00:16:36,760 --> 00:16:38,800 Over the last few weeks, I'm not going to name the outages. 256 00:16:38,900 --> 00:16:44,640 We've had some outages which could have been averted, either through proactive testing or by failure testing, right? 257 00:16:44,640 --> 00:16:49,680 So, yeah, I can't really answer what keeps you up at night. 258 00:16:50,020 --> 00:16:52,920 The infrastructure and the scale at which we operate keeps me up at night. 259 00:16:53,220 --> 00:16:56,440 But Cloud Foundry, in general, is pretty resilient in this matter. 260 00:16:56,620 --> 00:17:04,180 Like, you know, the whole concept of self-healing that Cloud Foundry brings in, like, you know, it's pretty good in that aspect. 261 00:17:04,400 --> 00:17:06,520 When something goes down, it brings it back. 262 00:17:06,520 --> 00:17:10,980 But there are some other areas, like storage, that Cloud Foundry depends on. 263 00:17:11,960 --> 00:17:15,140 That is where the resiliency and the soft spots are in the architecture. 264 00:17:15,400 --> 00:17:17,020 So, we have to keep an eye on that. 265 00:17:17,460 --> 00:17:17,820 Interesting. 266 00:17:17,980 --> 00:17:23,120 Is that a matter of those parts of the infrastructure just getting up to speed? 267 00:17:23,240 --> 00:17:26,680 I know that's been on the work and storage, but it's still kind of, like, it's in storage, right? 268 00:17:26,900 --> 00:17:30,460 Yeah, so for us, there's, you know, we've grown so quickly. 269 00:17:30,600 --> 00:17:36,220 So, there are some single point of failures in our architecture, a lot of which we've addressed last year, a lot we're going to address this year. 270 00:17:37,560 --> 00:17:46,040 And as we address these single points of failure, it's not just fixing it, but also validating what we've actually deployed, right? 271 00:17:46,360 --> 00:17:54,180 The S3 Blob Store, for example, that's behind our infrastructure, which is responsible or a key element of all the CF push enablement, right? 272 00:17:54,620 --> 00:17:56,320 We've run into this space issues with that. 273 00:17:56,740 --> 00:17:58,920 So, we've done a lot of re-architecture there. 274 00:17:59,340 --> 00:18:01,120 But oftentimes, we do the re-architecture. 275 00:18:01,220 --> 00:18:04,860 Are we running any load testing or are we running any failure testing on that infrastructure, right? 276 00:18:04,900 --> 00:18:07,080 That's the piece that I want to aspire towards this year. 277 00:18:07,440 --> 00:18:09,320 How can I validate that this is rock solid? 278 00:18:09,840 --> 00:18:16,160 Those are the things that we should be focusing on next, not just re-architecture, but running actual tests to validate our assumptions. 279 00:18:16,860 --> 00:18:21,080 Do the challenges come more from, as you grow, is it more from just the sheer scale? 280 00:18:21,280 --> 00:18:28,600 Is it, like, more services equals more issues or is it just, like, larger footprints, right? 281 00:18:28,800 --> 00:18:30,080 It's a combination of everything. 282 00:18:31,020 --> 00:18:36,900 If you look at our footprint, I mentioned some numbers today, 39,000 containers, 13 plus foundations. 283 00:18:37,840 --> 00:18:39,220 We're proud of some of these numbers. 284 00:18:39,340 --> 00:18:48,260 We're also not proud of some of the numbers because at the scale at which we're growing, we grew vertically to a point where some of our foundations are a nightmare to manage. 285 00:18:48,560 --> 00:18:49,660 I saw a talk on that. 286 00:18:49,800 --> 00:18:49,860 Yeah. 287 00:18:50,040 --> 00:18:52,500 And you saw Brandon's talk on how big is too big, right? 288 00:18:52,860 --> 00:18:58,880 There are challenges when you grow vertically and then you realize that your upgrades become very, very painful. 289 00:18:59,380 --> 00:19:02,520 Even simple repave operations are going to take quite a bit of time, right? 290 00:19:02,580 --> 00:19:09,140 So, we're addressing those by right-sizing foundations this year, building out more foundations, focusing on automation being the key element. 291 00:19:09,720 --> 00:19:18,560 In fact, one of the security vision talks about automation being, you know, a big driving force for how you deploy changes to your environments, right? 292 00:19:18,560 --> 00:19:22,080 It should be a modular, consistent, and repeatable process. 293 00:19:22,400 --> 00:19:33,120 It's been proven that in the information security space, anything that is automated has less error rate in terms of security. 294 00:19:33,860 --> 00:19:34,460 All right. 295 00:19:34,720 --> 00:19:39,860 I mean, you mentioned earlier, you know, obviously working with the application teams and you talked about game days and stuff. 296 00:19:40,560 --> 00:19:42,040 So, how does that work? 297 00:19:42,140 --> 00:19:47,460 I mean, how do you interact with developers and application teams whose applications? 298 00:19:47,460 --> 00:19:50,460 Yeah, there is an onboarding process. 299 00:19:50,640 --> 00:19:54,120 I'll slightly pressure upon it, but then I'll let Karun talk about the actual game day. 300 00:19:54,980 --> 00:19:56,980 Our product manager, Shipra, was also involved in it. 301 00:19:57,080 --> 00:19:58,960 She's a big driving force also, right? 302 00:19:59,020 --> 00:20:00,420 I'll talk about the high-level pitch. 303 00:20:00,780 --> 00:20:01,880 So, we had the toolkit ready. 304 00:20:02,160 --> 00:20:05,760 We came up with a process for onboards where we reached out to some of our key customers. 305 00:20:06,240 --> 00:20:08,900 We announced that we have a toolkit with which we want to run game days. 306 00:20:09,360 --> 00:20:13,060 And we gave them a perspective of what the game days is, what the onboarding process looks like. 307 00:20:13,100 --> 00:20:15,400 The onboarding process is answering some questions. 308 00:20:15,880 --> 00:20:21,080 In our questionnaire, we evaluate them and then we decide a date on when we want to meet them for the game days. 309 00:20:21,880 --> 00:20:26,120 And then once the game days run, we give them a prescription report of what else they could be doing better, right? 310 00:20:26,200 --> 00:20:27,820 Maybe you want to zoom in a little more on this. 311 00:20:28,100 --> 00:20:34,700 So, the first thing is we have a platform with 3,000 applications running and belonging to multiple customers. 312 00:20:35,200 --> 00:20:40,180 So, when we are conducting these game days, infrastructure-level chaos attack is ruled out. 313 00:20:40,540 --> 00:20:43,320 We didn't want to do it because it's going to impact several other customers. 314 00:20:43,320 --> 00:20:45,040 So, we have to get a justification from that. 315 00:20:45,500 --> 00:20:58,760 So, when we are doing this game day with a particular customer, we take their application, we find their, like, you know, how their design and architecture is, and take a wild guess, like, you know, fair guess, like, you know, what exactly may go wrong. 316 00:20:59,100 --> 00:21:01,160 So, we draft our experiments. 317 00:21:01,160 --> 00:21:05,500 So, when, on the day of game day, we conducted, like, two hours session. 318 00:21:05,980 --> 00:21:08,980 So, we announced the first experiment that we are going to conduct. 319 00:21:09,420 --> 00:21:13,380 And we expect the app team sitting on the other side of the table to take a guess. 320 00:21:14,140 --> 00:21:20,560 They have to investigate, detect, and also remediate what happened, how to fix that particular, you know, problem. 321 00:21:20,680 --> 00:21:27,640 So, let's say if we are injecting a failure test, they have to identify that this is the particular failure that we have triggered. 322 00:21:28,100 --> 00:21:28,200 Yeah. 323 00:21:28,300 --> 00:21:35,520 So, 10 out of the 10 plus teams you have reached, 6 have expressed interest, and we have a waiting list of people that want to go in with us, right? 324 00:21:35,520 --> 00:21:46,200 So, but there is equal amount of preparatory work that application teams will have to do in terms of getting the questionnaire filled up, having that design discussions with us, and then targeting a day for game day, right? 325 00:21:46,240 --> 00:21:47,980 What environments are we going to run the test on? 326 00:21:48,060 --> 00:21:50,620 So, we've been able to successfully do that with one team. 327 00:21:51,020 --> 00:21:56,380 But moving into Q2, like Karun said, we want to, you know, repeat rinse this test with a few other teams. 328 00:21:56,480 --> 00:22:05,760 In fact, I want to actually push some failure injection tests to the infrastructure level, but that will need a little more of a buy-in and sign-off from my own team, right? 329 00:22:05,800 --> 00:22:14,500 Because we consider our non-prod environment such as prod, so it's not as easy as just going and launching a network latency attack on a non-prod environment, because they're going to impact several hundred application teams. 330 00:22:14,860 --> 00:22:15,100 Right. 331 00:22:15,260 --> 00:22:17,660 Yeah, that seemed to be the challenge. 332 00:22:18,000 --> 00:22:27,240 Is there, like, how widely, like, I mean, how many application teams does the company have, right, and is the goal to hit this? 333 00:22:27,420 --> 00:22:32,240 I mean, I guess it depends on how fast CF adoption and PKS adoption happens, right? 334 00:22:32,660 --> 00:22:35,140 Yeah, the adoption for CF has been monumental. 335 00:22:35,440 --> 00:22:36,840 PKS is getting there. 336 00:22:37,060 --> 00:22:40,740 So, I think from an adoption standpoint, that's not a challenge. 337 00:22:40,860 --> 00:22:46,000 The challenge really has been, there is the what about the platform and how to use the platform. 338 00:22:46,380 --> 00:22:48,700 The challenge has been how to use the platform, right? 339 00:22:48,760 --> 00:22:51,780 Because if you know how to use the platform, you can build better resilient apps. 340 00:22:51,920 --> 00:22:57,460 That's the challenge that we're having at T-Mobile, and I don't necessarily see adoption as a challenge. 341 00:22:57,580 --> 00:22:59,880 Everybody wants to use our platform, to be honest with you. 342 00:23:00,620 --> 00:23:05,940 So, it's just coaching them on how to use the platform, and we're considering this as an opportunity to teach them how to use the platform. 343 00:23:06,700 --> 00:23:11,860 Earlier on, Karun mentioned on the 12-factor principles, which is conformance to standards of building right applications, right? 344 00:23:12,560 --> 00:23:18,300 A lot of the times, we see that application teams have their priorities around getting their code pushed to prod sooner, 345 00:23:18,540 --> 00:23:22,280 which means they may have to bypass some of the constraints on conformance to standards, 346 00:23:22,780 --> 00:23:25,920 which is essentially signing up for tech debt later on when an outrage happens. 347 00:23:26,720 --> 00:23:34,080 Just doing these targeted attacks with us or, you know, with these game day sessions with us will help them mature before they run their apps in production at scale. 348 00:23:34,280 --> 00:23:40,780 All right. How good do you think developers can actually get in terms of designing for resiliency, right? 349 00:23:40,900 --> 00:23:43,680 Because, I mean, there's no perfect, obviously, right? 350 00:23:43,720 --> 00:23:52,160 So, is there a point where, like, a level of, I don't know, adequacy might be the word, that would be the baseline? 351 00:23:52,420 --> 00:23:55,140 So, that's where this 12-factor comes into picture. 352 00:23:55,340 --> 00:24:02,060 So, that's the reason, like, you know, when you're designing an application, go through the standards and make sure you're compliant with it. 353 00:24:02,520 --> 00:24:07,680 And if you're following, at least, I'm sure, like, 60 to 70% of your application is resilient by default. 354 00:24:08,100 --> 00:24:10,800 And Cloud Foundry will fill in the rest of the gap. 355 00:24:11,060 --> 00:24:15,840 But still, there could be some external dependencies with the apps running inside the Cloud Foundry. 356 00:24:16,240 --> 00:24:20,880 Let's say if it is talking to an external database, what happens if the external database is down? 357 00:24:21,100 --> 00:24:22,740 You know, how is your application going to behave? 358 00:24:22,960 --> 00:24:36,820 So, those are the external factors that application team has to keep an eye on and have the resiliency designed for it, both from the, in a holistic way, not just for the apps that are in Cloud Foundry, but also for the apps that it is dependent on. 359 00:24:37,160 --> 00:24:37,300 Yeah. 360 00:24:37,660 --> 00:24:44,120 I have, I think T-Mobile has really, some really sharp developers that, because we've been interacting closely with them, right? 361 00:24:45,040 --> 00:24:51,080 But, again, like I mentioned, the challenge is going to be achieving that balance between their application development versus building resilient apps. 362 00:24:51,100 --> 00:24:53,900 And achieving the challenge is going to be a tough one, right? 363 00:24:54,020 --> 00:24:55,440 It is a culture change. 364 00:24:55,500 --> 00:25:05,220 Just like adopting a new platform is a culture change, the DevOps culture change I'm talking about, this is going to be yet another culture change, which is going to be something that is going to be done slowly but surely and steadily. 365 00:25:06,540 --> 00:25:10,840 And where I was going with that, I'm trying to miss the point now where I'm going with that. 366 00:25:10,840 --> 00:25:14,060 So, actually, it's not a one-stop solution for all the applications. 367 00:25:14,220 --> 00:25:15,340 So, every application is different. 368 00:25:15,760 --> 00:25:19,640 Every application that's deployed inside the PCF and PKS has their own needs. 369 00:25:20,000 --> 00:25:26,060 So, yes, we leave it to the developers to decide, and all the architects have a key role to play in that. 370 00:25:26,300 --> 00:25:36,720 Do you think, do you feel like you have or should have control over, or maybe I'm trying to get, maybe there's a framework that gets in place around these external pieces, right? 371 00:25:36,780 --> 00:25:38,060 I mean, the database, right? 372 00:25:38,200 --> 00:25:38,320 Yeah. 373 00:25:38,500 --> 00:25:42,360 These other pieces, because, obviously, those are going to become a major failure point, too. 374 00:25:42,480 --> 00:25:44,800 Yeah, you're talking about, like, a governance board that reviews architecture? 375 00:25:45,040 --> 00:25:53,500 Or, yeah, or just as, you know, a standard, like, short of some, like, the presumption you're deploying with this database. 376 00:25:53,500 --> 00:25:53,960 Sure, sure. 377 00:25:54,140 --> 00:26:02,700 Yeah, most of the teams have principal engineers and a member of technical staff embedded in their application team, so, who make a lot of really good design-driven decisions. 378 00:26:03,520 --> 00:26:09,480 We're being closely collaborating with some of the top, you know, minds at T-Mobile to form that governance body or center of excellence. 379 00:26:09,960 --> 00:26:19,300 In fact, there's some driving conversations already to create that center of excellence, which, you know, teams that are going through the microservice adoption will go through the center of excellence for some design patterns and analysis. 380 00:26:19,300 --> 00:26:23,460 Yeah, so, one thing we, as a platform engineering team, we do not do any more. 381 00:26:23,500 --> 00:26:25,840 We're focusing on how the apps are getting deployed. 382 00:26:26,140 --> 00:26:32,780 We can only educate, but there is design reviews that happens at the security end. 383 00:26:33,280 --> 00:26:40,920 So, but, again, it depends on every application that has to go to the production has to go through the process of design reviews within the team. 384 00:26:41,160 --> 00:26:50,600 So, that's why I'm curious to say, like, you know, you talked earlier about Netflix and some of the companies that pioneered chaos engineering. 385 00:26:50,720 --> 00:26:59,860 How much do you think you can actually, and you touched on this, how much can you actually learn from those companies when you, you know, when you go read, like, a Netflix blog post and they go in there, go see a talk? 386 00:27:00,300 --> 00:27:07,620 You know, or how much of it really is, like, it's bespoke to T-Mobile just because the platform and the application, they're so different. 387 00:27:07,960 --> 00:27:08,160 Yeah. 388 00:27:08,820 --> 00:27:10,380 Again, for me, it's a culture change, right? 389 00:27:10,440 --> 00:27:11,840 It has to be done slowly. 390 00:27:12,580 --> 00:27:20,380 If I go backwards in time to March or April of 2018, this was just getting started and we had no idea what we wanted to do. 391 00:27:20,580 --> 00:27:22,140 So, right, and then we got out on the right track. 392 00:27:22,480 --> 00:27:23,900 Today, we are here talking in front of you. 393 00:27:24,000 --> 00:27:25,060 We just did our first keynote. 394 00:27:25,300 --> 00:27:26,580 We had our deep divest today. 395 00:27:27,060 --> 00:27:30,160 So, we're getting the momentum or getting the advertisement done. 396 00:27:30,860 --> 00:27:33,020 We have customers lined up with us to do game day. 397 00:27:33,140 --> 00:27:37,420 So, I think the adoption is going to be steady as more people adopt PKS and PCF. 398 00:27:39,200 --> 00:27:42,280 Chaos engineering is a known concept to all development teams. 399 00:27:43,240 --> 00:27:45,480 What's not known is the framework that we have created. 400 00:27:45,720 --> 00:27:47,300 So, we're advertising the framework now. 401 00:27:47,440 --> 00:27:50,980 And I think the adoption for, you know, fast forwarding one year from now is going to be much bigger. 402 00:27:51,100 --> 00:27:56,240 And I can guarantee you that there will be at least 15 different game day sessions we might have done by then, right? 403 00:27:56,320 --> 00:27:58,600 And this is probably a larger conversation at that point. 404 00:27:59,320 --> 00:28:00,100 Think about Netflix. 405 00:28:00,220 --> 00:28:02,360 Netflix started this in 2011 or early 2010. 406 00:28:02,880 --> 00:28:07,680 And they were in the sweet spot because from the get-go, they started looking at AWS as their infrastructure. 407 00:28:07,840 --> 00:28:09,080 AWS was just getting started. 408 00:28:09,200 --> 00:28:10,440 So, it was a lot of partnership. 409 00:28:10,800 --> 00:28:19,260 And their history is also important here because people that went from Amazon to Netflix to create their Chaos Kong or Chaos Monkey framework, 410 00:28:19,780 --> 00:28:23,980 or people that worked at Amazon that knew failure testing and took the practices to Netflix. 411 00:28:24,340 --> 00:28:29,820 So, they were trying to get their culture going from the get-go that this is the right way to do it, right? 412 00:28:30,060 --> 00:28:35,760 For us, we're a little backwards here because we have a lot of legacy applications that are going through this transformation phase. 413 00:28:36,060 --> 00:28:40,340 So, I'm fairly confident that if we get to a point where we run these game days with a lot more teams, 414 00:28:40,860 --> 00:28:44,640 you know, we'll get to a point where we'll have much more better resilient apps in production. 415 00:28:45,060 --> 00:28:45,160 Right. 416 00:28:45,240 --> 00:28:53,420 I think that's the thing that gets overlooked sometimes is you read and you see, well, Netflix, like, so obviously, it could be easy for us, 417 00:28:53,420 --> 00:28:57,700 except, right, there's legacy, there's culture, there's all these things, there's institutional knowledge. 418 00:28:57,820 --> 00:28:57,900 Yeah. 419 00:28:58,020 --> 00:29:00,360 There's all these things that, like, most companies just don't have. 420 00:29:00,380 --> 00:29:01,500 And then there's leadership buy-in also. 421 00:29:01,640 --> 00:29:02,820 That's extremely important, right? 422 00:29:02,860 --> 00:29:06,600 Because if you don't have that buy-in, then you'll go nowhere with these things, unfortunately. 423 00:29:06,960 --> 00:29:09,920 So, the important thing is we know what our applications are. 424 00:29:10,040 --> 00:29:15,500 We have 80% of our applications built in Spring Boot framework, so Java and Spring Boot. 425 00:29:15,960 --> 00:29:22,300 So, whatever the toolkit that we are coming up with, it is generic enough, but it is more focused on the Spring applications as well. 426 00:29:22,720 --> 00:29:22,940 All right. 427 00:29:23,020 --> 00:29:30,560 And then I just want to, you know, kind of in closing here, I want to ask, is there, I mean, we talk, you know, I guess I tend to think of PCF in broad terms. 428 00:29:30,560 --> 00:29:33,320 And obviously, you guys are running on PAS and PKS. 429 00:29:33,480 --> 00:29:40,040 Is there, do you have to look at things differently if you're talking about applications or services running on PAS versus PKS? 430 00:29:40,300 --> 00:29:44,100 Are there, I would imagine, architecturally, right, there are just different gotchas out there. 431 00:29:45,280 --> 00:29:47,320 I'll take a stab, and then I'll let Karun know. 432 00:29:47,700 --> 00:29:48,680 PCF versus PKS, right? 433 00:29:48,680 --> 00:29:48,820 Yeah. 434 00:29:49,160 --> 00:29:58,200 So, before any customer is getting onboarded, we try to look at their application and try to divert them into the right platform 435 00:29:58,200 --> 00:30:00,600 that the application can be hosted on. 436 00:30:01,020 --> 00:30:09,320 So, if, for example, if the application that's been developed is a Spring Boot and which has a dependency with the config server, service registry, 437 00:30:09,480 --> 00:30:11,240 then, you know, obviously go with PCF. 438 00:30:11,700 --> 00:30:14,680 PKS is not going to help them out in that aspect. 439 00:30:15,180 --> 00:30:23,340 But in cases where customers are coming up with their own container, like Docker containers, then, obviously, PKS is their ideal choice. 440 00:30:23,900 --> 00:30:28,960 I think the other follow-up question you had was, is chaos engineering different for apps built on PCF and PKS? 441 00:30:29,040 --> 00:30:31,500 Not really, because under the hood, it's all Bosch-controlled and deployed. 442 00:30:31,980 --> 00:30:35,980 For us, it's one framework that should work seamlessly for apps built on PCF and PKS. 443 00:30:36,660 --> 00:30:38,240 Because our infrastructure is the same for us. 444 00:30:38,240 --> 00:30:42,680 So, again, the chaos toolkit that we are building is more specific to Bosch-deployed environment. 445 00:30:42,680 --> 00:30:42,920 Okay, yeah. 446 00:30:42,920 --> 00:30:48,760 So, since PCF and PKS are both backed with Bosch, the Turbulence++ is a toolkit that we have. 447 00:30:48,960 --> 00:30:52,180 We have an engineer, Matthew, who has been extensively working on this. 448 00:30:52,320 --> 00:30:54,240 Conor Matthew is part of my team. 449 00:30:54,380 --> 00:31:02,520 So, we have plans of trying to deploy this Turbulence++ and Monarch into the PKS environment in the near future. 450 00:31:02,860 --> 00:31:09,480 Okay, and then just to close, guys, I mean, you've touched on some of this already, so maybe don't feel free not to repeat yourself too heavily. 451 00:31:09,480 --> 00:31:17,360 But, I mean, you mentioned, like, okay, you got, like, a handful of one team or a handful of teams, you know, bought in and ready. 452 00:31:17,880 --> 00:31:23,960 I mean, what is, what are the end goals, I guess, for the chaos engineering aspect of your job? 453 00:31:24,660 --> 00:31:34,780 For me, it's, if these fields, like, the team with which we ran game days, this is a team that does coverage map analysis, and it's a front-facing app. 454 00:31:34,920 --> 00:31:39,740 We ran attacks, several kinds of attacks, simulated attacks in their non-prod environment. 455 00:31:40,340 --> 00:31:47,160 If they feel comfortable that their app is mature, resilient, and if they are willing to bet that let's target these attacks in prod, that's the big winner for me, right? 456 00:31:47,240 --> 00:31:54,240 But it's really them taking our prescriptions, making sure that they feel comfortable that they can run these attacks in production. 457 00:31:54,740 --> 00:31:59,180 In fact, this team is running their workloads in PCF, PKS, and also in the public cloud. 458 00:31:59,500 --> 00:32:06,180 So, it's a great team to run such attacks because we could take, we could run our attacks in one and around one and see how it impacts the customers. 459 00:32:06,640 --> 00:32:07,740 That's the desired state. 460 00:32:08,300 --> 00:32:26,660 So, oftentimes, like, you know, if teams are a little shy coming to the game days, we also wanted to come up with a concept of providing it as a service, wherein developers or the development teams can use that as a service, chaos engineering as a service, and perform their attacks as and when they want. 461 00:32:26,800 --> 00:32:36,180 Yeah, so part of that, what Karun's talking about is, you know, your continuous integration and deployment pipeline has automated tests as part of your CICD as code goes from one stage of the pipeline to another, right? 462 00:32:36,200 --> 00:32:40,720 So, what he and I are envisioning is why not just have failure testing as part of the CICD? 463 00:32:40,820 --> 00:32:42,100 So, it happens in a lower end environment. 464 00:32:42,340 --> 00:32:50,740 So, as code matures and goes to Prague, this chaos engineering as a service will target these failure tests, and good quality code gets pushed to production. 465 00:32:50,980 --> 00:32:51,520 Oh, that's clever. 466 00:32:51,640 --> 00:32:53,140 So, we're just taking out that extra. 467 00:32:53,500 --> 00:32:53,760 Yes. 468 00:32:53,860 --> 00:32:54,140 That extra stuff. 469 00:32:54,140 --> 00:33:04,360 So, rather than having them coordinate with us if they're shy, they're more than welcome to use our framework, integrate it with their test framework, and just integrate it with their pipelines, and just naturally have the code go through this sort of maturity. 470 00:33:04,840 --> 00:33:05,240 All right, cool. 471 00:33:05,280 --> 00:33:05,940 Yeah, that makes a lot of sense. 472 00:33:06,360 --> 00:33:07,040 Anything else? 473 00:33:07,280 --> 00:33:08,440 Anything we didn't touch on? 474 00:33:08,620 --> 00:33:09,420 No, this is great. 475 00:33:09,860 --> 00:33:10,800 A lot of great questions. 476 00:33:10,820 --> 00:33:11,620 Thank you for having us. 477 00:33:11,800 --> 00:33:12,060 All right. 478 00:33:12,120 --> 00:33:12,760 Thanks for being on, guys. 479 00:33:12,960 --> 00:33:13,500 Yeah, thank you.