1 00:00:00,240 --> 00:00:05,580 Welcome to another episode of Cloud and Culture. I'm Danielle Burrow from VMware. 2 00:00:06,080 --> 00:00:11,000 And I'm Derek Harris from VMware. If you're new, this podcast focuses on the process of 3 00:00:11,000 --> 00:00:16,540 application IT modernization through the lens of Tanzu Labs, previously called Pivotal Labs. 4 00:00:17,020 --> 00:00:20,860 Over the years, Tanzu Labs experts have helped hundreds of companies and organizations, 5 00:00:21,220 --> 00:00:25,020 big and small, build better software and adopt modern development practices. 6 00:00:25,240 --> 00:00:30,100 The tradition carries on today as Tanzu Labs experts help clients to get up to speed with 7 00:00:30,100 --> 00:00:34,120 cloud-native computing and the flexibility and resiliency it provides. 8 00:00:34,620 --> 00:00:40,400 Our guest for this episode is Sean Carey, the Minister of Chaos, who knows all about resiliency. 9 00:00:41,040 --> 00:00:44,720 More specifically, as his title suggests, he knows all about chaos engineering. 10 00:00:45,240 --> 00:00:49,360 He walks us through the evolution of chaos engineering from its modern inception at places 11 00:00:49,360 --> 00:00:53,280 like Netflix through today, when it's practiced by all sorts of organizations. 12 00:00:53,600 --> 00:00:58,160 He also gives some great advice on techniques for chaos engineering at various steps through 13 00:00:58,160 --> 00:01:03,640 the journey and discusses how the practice relates to other trends, such as SRE, and acceptance 14 00:01:03,640 --> 00:01:06,060 of failure as a tool for learning and improvement. 15 00:01:06,640 --> 00:01:08,560 Yeah, it's definitely an informative discussion. 16 00:01:08,740 --> 00:01:13,060 So I suggest if you've already made it this far, you stick around and learn from Sean why you 17 00:01:13,060 --> 00:01:16,460 should be paying attention to chaos engineering if you're not already doing it. 18 00:01:17,280 --> 00:01:18,080 Hi, Sean. 19 00:01:18,180 --> 00:01:19,640 Thanks so much for joining us today. 20 00:01:19,820 --> 00:01:25,600 Could you just give us a really quick introduction of who you are and what you do at Tanzu Labs? 21 00:01:26,680 --> 00:01:27,460 Hi, Danielle. 22 00:01:27,580 --> 00:01:28,460 Thanks for having me. 23 00:01:28,760 --> 00:01:30,180 My name is Sean Carey. 24 00:01:30,440 --> 00:01:39,140 I am the Minister of Chaos at Tanzu Labs within the Modern Application Platform Business Unit. 25 00:01:40,140 --> 00:01:45,300 I deliver chaos to all of my customers. 26 00:01:46,480 --> 00:01:47,080 Love it. 27 00:01:47,460 --> 00:01:52,300 When you say you're the Minister of Chaos, what does that really mean? 28 00:01:52,500 --> 00:01:53,820 What does that mean? 29 00:01:53,900 --> 00:01:55,660 Dig into that for us a little bit, Sean. 30 00:01:56,640 --> 00:02:06,240 What I do is I help customers understand how chaos engineering and fault injection can improve the reliability of their systems. 31 00:02:06,240 --> 00:02:06,840 Okay. 32 00:02:07,300 --> 00:02:15,640 And when we talk about that, obviously, that's more broadly known as chaos engineering for the uninitiated, right? 33 00:02:15,660 --> 00:02:22,140 Or for anyone who's heard of chaos engineering but hasn't really dove into it or taken the time to understand it. 34 00:02:22,200 --> 00:02:25,420 I mean, what are we talking about when we're talking about chaos engineering? 35 00:02:25,540 --> 00:02:30,420 What are some of the practices and what's the overall thing you're trying to accomplish with it? 36 00:02:31,180 --> 00:02:42,860 Chaos engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production. 37 00:02:43,360 --> 00:02:50,500 It allows us to identify weaknesses before they manifest in system-wide aberrant behaviors. 38 00:02:51,220 --> 00:03:07,160 Netflix coined the term, and then Google has a very similar practice called fault engineering, where they inject errors into their systems to ensure that they're handling them appropriately. 39 00:03:07,780 --> 00:03:08,280 Okay, great. 40 00:03:08,280 --> 00:03:14,660 And can you, I would love to get an example of like, what are these errors? 41 00:03:14,880 --> 00:03:18,340 Like, what are these things that the systems are needing to tolerate? 42 00:03:18,640 --> 00:03:22,220 What kinds of chaos are you subjecting people to? 43 00:03:22,220 --> 00:03:33,800 Well, Netflix started by shutting down machine instances to see how they affect overall performance of their content delivery network. 44 00:03:34,060 --> 00:03:47,560 In Google's case, what they do is inject bad data into a web request, or they can take that web request and just throw it away. 45 00:03:47,560 --> 00:03:57,500 And what is the, like, but I guess I think, and hopefully I think, Danielle, where you're getting at, is like, what, I mean, we're sort of testing out basically, you know, the goal here is resilience, right? 46 00:03:57,700 --> 00:04:02,560 The goal is to make sure, or, you know, if it's not resilience, it's something along those lines, correct? 47 00:04:02,920 --> 00:04:14,020 Like, to make sure, yeah, if a machine goes down, if the whole cloud region goes down, if you're getting bad requests that the application can actually stay online and functioning, correct? 48 00:04:14,020 --> 00:04:19,880 Correct. So, I like to think of it as an extension to test-driven development. 49 00:04:20,340 --> 00:04:24,640 We start with our unit tests of the actual package we're developing. 50 00:04:25,140 --> 00:04:28,900 Then we do an integration test with the systems we know it's interacting with. 51 00:04:29,520 --> 00:04:34,780 And then chaos engineering provides us a larger scale testing platform. 52 00:04:34,780 --> 00:04:44,640 And what does that look like for, when you talk about this, for customers, for organizations, like, what, how does one even get started going down this road? 53 00:04:44,720 --> 00:04:46,260 I mean, are there other tools they can use? 54 00:04:46,300 --> 00:04:54,760 Is it just, I mean, obviously it's not a matter of just randomly shutting down your instances and hoping for the, your production instances and hoping for the best, right? 55 00:04:54,800 --> 00:04:59,520 So, kind of, can you walk us through the process of even getting started with chaos engineering? 56 00:04:59,520 --> 00:05:04,280 Yeah, most customers getting started, what we do with them is we hold a game day. 57 00:05:05,160 --> 00:05:12,100 So, we ask them to think up, and when I say they, I mean the platform team generally, right? 58 00:05:12,100 --> 00:05:19,860 The people who are operating the underlying compute storage and networking space. 59 00:05:20,160 --> 00:05:34,940 And we say, what could you possibly do to your system that would impact its availability to the customer or its security to, of, of the customer data? 60 00:05:35,540 --> 00:05:44,120 So, we take those game days, we have people sit down in a room and start brainstorming scenarios that could have impacts. 61 00:05:44,580 --> 00:05:53,340 And then the second half of the day would be picking the scenarios that we think would limit the blast radius. 62 00:05:53,440 --> 00:05:59,080 We don't want to just shut down our Cisco routers because that's going to take down our entire system. 63 00:05:59,180 --> 00:06:05,780 So, we talk about blast radius and say, how can we do a small experiment and measure that impact? 64 00:06:06,500 --> 00:06:13,560 So, once we do that, then we identify what we need to be measuring. 65 00:06:14,140 --> 00:06:21,800 So, when you talk about tools, really the most important tool to have to get started with is some sort of observability tool, right? 66 00:06:22,040 --> 00:06:40,280 Ideally, one that can look at your applications, your network, and your compute resources to try and identify whether your experiment has actually caused reliability to decrease over the system. 67 00:06:41,220 --> 00:06:42,880 That would be a really great way to start. 68 00:06:43,040 --> 00:06:45,680 Obviously, there's a lot of other tools that we can talk about. 69 00:06:45,680 --> 00:06:52,880 But the chaos monkey was the first tool that Netflix put out there. 70 00:06:53,380 --> 00:06:57,980 And it was designed to allow developers to shut down their own systems, right? 71 00:06:58,060 --> 00:06:59,340 So, it's self-service chaos. 72 00:06:59,920 --> 00:07:03,480 This is a little more mature chaos. 73 00:07:04,880 --> 00:07:07,660 I'm guessing that depends based on the organization, huh? 74 00:07:07,740 --> 00:07:13,640 Like Netflix back in 2010, different story than Chase today, for example. 75 00:07:14,420 --> 00:07:17,720 Self-service chaos is a great phrase, Sean. 76 00:07:17,960 --> 00:07:18,600 I love that. 77 00:07:19,180 --> 00:07:31,280 So, yeah, I mean, speaking of Netflix, can you talk a little bit about how chaos engineering has kind of evolved since, you know, it first kind of began in these big tech companies? 78 00:07:32,480 --> 00:07:32,600 Yeah. 79 00:07:32,700 --> 00:07:41,060 So, chaos engineering has moved beyond Netflix because they were focused on the infrastructure that they were running on, 80 00:07:41,060 --> 00:07:43,380 which was Amazon EC2. 81 00:07:43,660 --> 00:07:51,660 And really, we've seen an extension into all different kinds of software and hardware, right? 82 00:07:51,700 --> 00:07:53,160 I think I talked about it earlier. 83 00:07:53,360 --> 00:07:55,400 Security is very important. 84 00:07:55,540 --> 00:08:09,560 What happens if we start messing with groups, permissions, denying people permissions, denying systems permissions, networking, slowing down requests, throwing out requests, changing requests. 85 00:08:09,560 --> 00:08:14,200 This has become pretty popular. 86 00:08:14,200 --> 00:08:22,340 The other big piece that the game days add a lot to is for training new people, right? 87 00:08:22,600 --> 00:08:39,440 So, if you have a scenario that you can reproduce on a regular basis through automation and the use of tools, as Derek talked about, we can bring our new hires into a room and say, okay, the system is broken, fix it. 88 00:08:39,440 --> 00:09:00,080 So, we can understand what it is they actually need to do to get it fixed and help them along that path in learning about how they are going to operate the system in the future and then potentially provide additional avenues for fault injection chaos engineering in the future. 89 00:09:00,080 --> 00:09:07,680 And you mentioned, you mentioned like that, you know, it started with Netflix and Netflix was running solely or at the time mostly on Amazon. 90 00:09:08,420 --> 00:09:14,800 What, how important is like a cloud base or how important is the platform to doing chaos engineering? 91 00:09:14,900 --> 00:09:23,080 Because it seems like something that wasn't really feasible in a pre-cloud or at least pre-cloud native sort of, sort of environment. 92 00:09:23,180 --> 00:09:26,180 Is that accurate or like any chaos engineer on, you know, way? 93 00:09:26,180 --> 00:09:35,480 I mean, I've been a chaos engineer long before the term was invented because I worked in a cramped data center and I used to knock cables out all the time. 94 00:09:36,380 --> 00:09:41,100 So, if you don't have a cloud, yeah, go pull some network cables, go pull some power cables. 95 00:09:41,880 --> 00:09:43,380 That's definitely an option. 96 00:09:45,060 --> 00:09:46,600 Seems a little less controllable. 97 00:09:46,960 --> 00:10:07,060 But it's a little less controllable and you cannot provide that level of self-service that we were talking about, which allows that blast radius to be minimized and gives the development teams, both platform and application, the opportunity to create new experiments without having bottlenecks. 98 00:10:07,900 --> 00:10:11,180 And this is kind of a tangent, but, or, but it's related. 99 00:10:11,340 --> 00:10:25,880 So, I'm just curious when you talk about tripping over cables and stuff, the one, it just does strike me that the one thing you often see if there's a big outage a lot of times, even though all these companies are doing chaos engineering, the post-mortem on the outage, sometimes it just comes down to a human mistake, right? 100 00:10:25,940 --> 00:10:28,560 Like that was not game plan for presumably. 101 00:10:29,420 --> 00:10:44,580 Yeah, and that's, you know, 70% of those errors are just someone typing something incorrectly, which is why we at Tanzu Labs love our pair programming, because we can see those mistakes before they make it to production. 102 00:10:45,220 --> 00:10:50,300 Well, it sounds like chaos engineering and fault injection is very necessary. 103 00:10:50,300 --> 00:11:00,680 Like when you're describing all the, all the things that you're doing and all the, you know, the, the ways you're watching the system respond and, and helping, you know, using it also to facilitate kind of training. 104 00:11:00,820 --> 00:11:03,660 It sounds like more than just kind of a nice to have. 105 00:11:04,120 --> 00:11:08,000 And I wonder, you know, what's the adoption look like out in the enterprise? 106 00:11:08,000 --> 00:11:14,000 Is this a common practice or is this something that folks are, you know, kind of struggling to adopt? 107 00:11:14,120 --> 00:11:17,760 Like where, where is kind of your average enterprise? 108 00:11:18,780 --> 00:11:24,460 Our average enterprise would be very, very early adoption, right? 109 00:11:24,460 --> 00:11:50,000 There's, there's, there's still a fear in a lot of organizations because they have very strict change control processes and change advisory boards and fun things like that to allow experimentation in production environments, which is really where you're going to get the most benefit of it. 110 00:11:50,820 --> 00:12:04,280 Obviously we, as I talked about blast readiness earlier, we try and start our experiments in a sandbox where if we do take something down, say on the platform side, there are no developers using that platform. 111 00:12:04,280 --> 00:12:21,220 And then we recommend that the platform users or application developers test in a non-production environment first, just as they've done historically with low testing and performance testing practices before they move into production. 112 00:12:21,700 --> 00:12:36,960 And is there a point at which it makes sense to get started because I guess the difference, like when I think about a Netflix or Google versus an enterprise application, a lot of, you know, the differences are in terms of scale and number of users and, and traffic and all these things. 113 00:12:37,020 --> 00:12:54,820 I mean, is there a point where, and, you know, probably architecture, is there a point where it makes sense, like it almost becomes a necessity to start doing this, whether it's in terms of adopting microservices or if you're, you know, your traffic starts picking up markedly or whatever it is where you really, really need to start doing chaos engineering. 114 00:12:54,820 --> 00:13:15,460 My background is in financial services and what I found was a great way to encourage people to understand where it's going to add value quickly is by measuring the effect of downtime of your most critical system, right? 115 00:13:15,940 --> 00:13:20,720 And if, when we looked at ours, it was about $50,000 a second. 116 00:13:21,200 --> 00:13:23,360 So about $3 million a minute. 117 00:13:24,300 --> 00:13:27,360 So it made sense to invest, right? 118 00:13:27,380 --> 00:13:32,300 If it's one engineer at say $100,000 a year, right? 119 00:13:32,300 --> 00:13:41,200 All their time spent on stopping this system from going down would be a 30 times return on investment. 120 00:13:42,480 --> 00:13:44,220 So that one, that one pays off. 121 00:13:44,760 --> 00:13:45,040 Okay. 122 00:13:45,280 --> 00:13:45,780 So yeah, exactly. 123 00:13:45,840 --> 00:13:46,320 That's kind of it. 124 00:13:46,380 --> 00:13:48,420 I mean, it's, it's value basically, right? 125 00:13:48,440 --> 00:13:53,240 I mean, if you have, if I'm an organization, I say, if this thing goes down, it's bad, but whatever. 126 00:13:53,380 --> 00:13:54,840 I mean, I guess, depending on the application. 127 00:13:54,840 --> 00:13:59,460 I would say, I would say the second piece, Derek, is, is security. 128 00:13:59,460 --> 00:14:06,840 Again, the average cost of a, of a system breach, according to the latest Verizon report is $3.8 million. 129 00:14:08,220 --> 00:14:18,540 So if you can potentially stop a breach occurring, by creating an experiment, which would minimize a configuration error in your credentials. 130 00:14:19,220 --> 00:14:21,100 Again, it's, it's value. 131 00:14:21,440 --> 00:14:27,940 The last piece I would talk about is your team, right? 132 00:14:29,220 --> 00:14:37,580 This reliability engineering practice that Google's put together, the, the goal of it is to reduce toil. 133 00:14:37,980 --> 00:14:44,520 Your operators, your developers, they don't, it doesn't add value for them to be doing the same thing over and over again. 134 00:14:44,840 --> 00:14:57,860 So for me, chaos engineering injects some fun into the job, go and try and break our systems for me as an engineer, as an architect, that's something I look forward to coming to work to do. 135 00:14:58,080 --> 00:15:01,640 And we had, we talked about SRE in a previous podcast, and that's a nice dovetail. 136 00:15:01,720 --> 00:15:04,740 We talked totally about, about reducing toil there too. 137 00:15:04,760 --> 00:15:07,480 And these two things are definitely connected, right? 138 00:15:07,940 --> 00:15:08,200 Yep. 139 00:15:08,500 --> 00:15:17,420 I mean, can you walk through, I mean, can, can you walk through, do you work, when you think of chaos engineering or when you're doing this in, in practice, is SRE and adopting some of those principles? 140 00:15:17,420 --> 00:15:21,000 Does that go hand in hand with, with, with what you're doing? 141 00:15:21,280 --> 00:15:21,760 Absolutely. 142 00:15:22,040 --> 00:15:31,340 You know, starting with the game days as an early adoption practice, what we can do is say, if we break this system, right? 143 00:15:31,440 --> 00:15:40,820 Say our payment system, will we actually find out that we've broken it before our customer calls us from a reliability engineering perspective? 144 00:15:41,940 --> 00:15:47,700 That's a measurement that I don't currently consume. 145 00:15:48,400 --> 00:16:00,000 So I would need to then go back and say, let's write some code to enter, to create a new alert that would let me know if I have this type of failure, right? 146 00:16:00,840 --> 00:16:09,160 So by even just having game days without actually creating any chaos, we can improve the reliability of our systems. 147 00:16:09,820 --> 00:16:11,360 Does that answer your question, Derek? 148 00:16:11,940 --> 00:16:14,100 Yeah, yeah, definitely. 149 00:16:14,720 --> 00:16:16,760 And you mentioned some tools previously. 150 00:16:16,940 --> 00:16:35,160 I'm wondering, you know, in terms of strategy and tools, like, or just looking at the larger system, like how much of your chaos engineering strategy is affected by, like, the teams, like their platform that they're using? 151 00:16:35,160 --> 00:16:37,680 Or maybe even the application architecture? 152 00:16:37,680 --> 00:16:47,320 Are there certain architecture choices that really make it easier to do this type of work? 153 00:16:47,320 --> 00:16:48,100 Definitely. 154 00:16:48,100 --> 00:16:56,800 The benefit of using a platform like Kubernetes, Tanzu Kubernetes Grid, is immense. 155 00:16:56,800 --> 00:17:10,620 It's the chaos mesh open source project allows you to create what are called custom resource definitions or operators in the Kubernetes world. 156 00:17:10,620 --> 00:17:20,880 And that gives you the ability to do that self-service chaos that's very valuable in a high-performing organization. 157 00:17:20,880 --> 00:17:32,220 It also allows you to extend the chaos beyond what comes out of the box, which is pretty comprehensive to things that are business domain specific. 158 00:17:32,220 --> 00:17:40,200 And it also allows you to give those extensions back to the community and continue to share open source software. 159 00:17:40,200 --> 00:17:48,500 But if you look back, I was actually just talking to another customer who has a huge web presence, right, that doesn't run in a containerized operation. 160 00:17:48,500 --> 00:17:55,540 And it's not quite a monolith, but there's a lot of pieces that can still be decomposed into microservices. 161 00:17:56,660 --> 00:18:15,620 And what we talked about with them was injecting their chaos through a proxy that would sit between a load balancer and that application internally and just take chaos-specific traffic from an internal machine that's just generating load. 162 00:18:15,980 --> 00:18:26,560 So you're protecting yourself from any downstream customer impact, but still able to add value on the fault injection front. 163 00:18:27,740 --> 00:18:28,660 And you mentioned Kubernetes. 164 00:18:28,820 --> 00:18:37,640 One of the things that we've talked about, I mean, for years now, I think, is this debate between whether Kubernetes is like a developer platform or an operational platform. 165 00:18:37,640 --> 00:18:38,720 So it's arguably some of both. 166 00:18:38,720 --> 00:18:50,780 But with regard to chaos engineering, do you see, I mean, where does the onus ultimately lie, I guess, in terms of, or do you see it often lie in terms of getting stuff? 167 00:18:50,900 --> 00:18:56,980 Is it in the application or is it, you know, on the operational side where your clients need to make improvements? 168 00:18:58,040 --> 00:19:07,740 Yeah, you know, having worked in Tanzu Labs on the platform team for a long time, I consider myself a software developer, even though I'm an operations guy. 169 00:19:08,460 --> 00:19:19,880 So the really cool thing about the chaos mesh, right, is that it can go down to the application level and create the chaos there. 170 00:19:20,560 --> 00:19:26,320 If you allow it, it's all role-based access control, so permissions to hit that application. 171 00:19:26,520 --> 00:19:32,920 Or you can go the other way and talk to your cluster API and say, hey, shut down this instance, right? 172 00:19:33,020 --> 00:19:34,080 Turn off this networking. 173 00:19:34,940 --> 00:19:37,640 So it's a platform for building platforms. 174 00:19:37,640 --> 00:19:47,820 And in this case, we've built a chaos platform, which allows us to impact both the operational platform and the application platform. 175 00:19:48,300 --> 00:19:50,840 I know I avoided your question a little bit there. 176 00:19:51,460 --> 00:19:52,440 Take a side. 177 00:19:53,640 --> 00:19:54,520 I know, I know. 178 00:19:54,580 --> 00:19:55,320 It's often both. 179 00:19:55,540 --> 00:19:58,860 It's often a gray area, but I just want to flesh out the topic. 180 00:19:58,860 --> 00:20:11,380 Yeah, and I think, I mean, I think you touched on this a little bit, Sean, when you were talking about kind of the ROI behind doing chaos engineering. 181 00:20:11,560 --> 00:20:17,700 But I'm wondering, you know, what some of the, like, objections are to doing it. 182 00:20:17,700 --> 00:20:26,840 And, you know, is it really just a matter of sitting down and kind of making those calculations with clients to get them to, you know, do it? 183 00:20:26,900 --> 00:20:34,580 Or is it, like, how do you work with folks who maybe part of the organization wants to do this and maybe other folks don't? 184 00:20:36,120 --> 00:20:40,920 Yeah, I think what Derek was just talking about, the separation between ops and apps. 185 00:20:40,920 --> 00:20:50,360 So we have a large retail customer who also, huge online presence, right? 186 00:20:50,440 --> 00:20:55,640 This year they had the fourth amount of traffic of any retailer on Black Friday. 187 00:20:56,640 --> 00:21:05,400 And so we said we'd like to try it on Black Friday in a very, very small part of the site. 188 00:21:05,400 --> 00:21:27,320 And then we started earlier in the summer building up to it and showing those application teams, the business owners, the ability to control our experiments when things went wrong, which is very important to have the big red button to be able to jam on and stop everything. 189 00:21:28,400 --> 00:21:40,460 But by building up to that, we actually ran into a production error that we were actually running the test a week before Black Friday. 190 00:21:41,160 --> 00:21:44,920 And we caused a default that caused downtime. 191 00:21:44,920 --> 00:21:53,480 Now we're estimating, because we don't have the business numbers, we caused $7 million worth of business impact. 192 00:21:54,420 --> 00:22:04,220 But looking at the traffic volume for Black Friday, we saved $63 million worth of downtime on Black Friday. 193 00:22:05,180 --> 00:22:14,740 So by showing them that we could fix the problems we had created very quickly, we were able to build up that confidence. 194 00:22:15,140 --> 00:22:22,980 And that's usually where people get uncomfortable is when you jump into production first and say, oh, well, we need to start shutting down AWS regions. 195 00:22:22,980 --> 00:22:24,280 They go, no, no, no, hold on a second. 196 00:22:24,380 --> 00:22:25,020 Wait, what are you doing? 197 00:22:25,520 --> 00:22:33,340 But if you can build up, right, that confidence level, show your maturity as an organization, the detractors tend to fade away. 198 00:22:35,180 --> 00:22:35,580 Yeah. 199 00:22:35,720 --> 00:22:37,160 And do your clients usually bring it up? 200 00:22:37,800 --> 00:22:43,440 Is chaos engineering something that you bring up to them or that companies would bring up to you as something they want to do? 201 00:22:43,440 --> 00:22:43,920 Yeah. 202 00:22:44,020 --> 00:22:49,520 I mean, with my title being Minister of Chaos, when I introduce myself, it gets them thinking about it. 203 00:22:50,520 --> 00:22:58,060 But as a topic, you know, we try and introduce it as part of our reliability engineering practice, right? 204 00:22:58,400 --> 00:23:03,060 When we see that they're at the maturity level where it's going to add value to their organization. 205 00:23:04,500 --> 00:23:08,320 And do, Sean, do some customers come seeking it, though? 206 00:23:08,480 --> 00:23:14,840 I mean, is that an area that, you know, say they know they want to get better at and they want to upskill their teams to be able to do it? 207 00:23:15,900 --> 00:23:25,080 Yeah, where we've seen more customers seeking it is actually on the application side because they're not familiar with it. 208 00:23:26,640 --> 00:23:35,080 And they've heard about some of our successes in other customers on the operational sides thanks to podcasts like this. 209 00:23:35,340 --> 00:23:41,280 I know we've heard customers like T-Mobile speak at SpringOne about their chaos engineering practice. 210 00:23:41,280 --> 00:23:48,900 And I wondered if there are certain industries that are more maybe ahead of the curve or more aggressively adopting these practices. 211 00:23:49,220 --> 00:23:52,340 Like, would you say telco is one of those industries and other others? 212 00:23:52,920 --> 00:23:56,360 Yeah, we're working with a customer who's, you know, building a 5G network. 213 00:23:57,260 --> 00:24:01,340 And just culturally, it's a new part of their business. 214 00:24:02,060 --> 00:24:08,400 And so what they're trying to instill in the people building that business is that failure is acceptable. 215 00:24:09,480 --> 00:24:25,360 So chaos engineering is something they wanted to do up front to get the people's mindset who are building this network wrapped around that, you know, failure is acceptable. 216 00:24:26,360 --> 00:24:37,980 And what we've seen is really it's a lot of high-volume organizations that are out in the lead, right, where 25% of your business is occurring on a single day. 217 00:24:38,840 --> 00:24:42,040 And you just, you can't have downtime. 218 00:24:42,660 --> 00:25:00,820 But realistically, right, with the complex microservices, Kubernetes, right, running 90 itself, as this complexity grows, right, if you do the math, right, the uptime gets harder and harder to keep where the business demands. 219 00:25:01,820 --> 00:25:18,280 And is that like, I was going to say, I mean, T-Mobile actually strikes me as an interesting company as well, because, like, you know, again, I mean, it's not like their traffic happens all at 25% of it happens in one day, for example. 220 00:25:18,280 --> 00:25:26,080 But like, when something happens, when you have whatever millions of consumers as users, for example, that's a different type of volume, I guess. 221 00:25:26,140 --> 00:25:33,600 You know, T-Mobile, I remember, I remember I did a podcast with them actually at SpringOne, like, it seems like forever ago now that we had a live SpringOne on. 222 00:25:34,060 --> 00:25:46,880 But it was about, you know, we talked about, like, when the CEO notices, when the CEO notices an outage, that was a pretty big indication that they needed to get on this and get on the reliability factor. 223 00:25:46,880 --> 00:25:51,540 So, I guess, yeah, I just, that's just wrong with me when you mentioned T-Mobile and the high volume, right? 224 00:25:51,680 --> 00:25:56,520 Like, obviously, they do lots of traffic a lot of times, but, you know, they just have a lot of users, period. 225 00:25:57,400 --> 00:26:04,140 Well, that, and the instance we worked with them initially to focus on was the iPhone launch. 226 00:26:05,220 --> 00:26:15,440 So, iPhone launches have become more mundane nowadays, but back in the day, right, that was, they sold a lot, a lot of phones, and they had a lot of downtime. 227 00:26:15,440 --> 00:26:20,200 I think the first three iPhone launch days, they had downtime. 228 00:26:20,900 --> 00:26:26,460 So, by the time we got our platform up and running, I think four through 11 are we on now? 229 00:26:26,780 --> 00:26:31,020 We haven't had any customer-facing downtime since then. 230 00:26:31,540 --> 00:26:31,800 Cool, yeah. 231 00:26:31,880 --> 00:26:39,080 And I think, yeah, the other one that comes out as a customer, that example that comes about, I don't know that it was, that you guys worked with them or that they had, 232 00:26:39,480 --> 00:26:45,820 and that they had worked on this was, I think it was Dick's Sporting Goods around, they had talked about, just on a similar thing to the iPhone launch. 233 00:26:45,920 --> 00:26:51,420 And I know in the Sporting Good world, there aren't a lot of analogs, but, like, new shoe launches can be one of those. 234 00:26:51,480 --> 00:26:55,780 I think it was, like, the Yeezy, this, I don't even know how old the story was, a year, year and a half ago. 235 00:26:55,880 --> 00:27:00,660 But, yeah, it was, like, they were able to go and do this Yeezy launch, which is a big deal if you're not a sneakerhead. 236 00:27:00,920 --> 00:27:01,940 That's apparently a big deal. 237 00:27:02,100 --> 00:27:02,660 I'm not one. 238 00:27:02,880 --> 00:27:07,680 But, and I was, like, yeah, I know they were talking online and tweeting about how they didn't have any downtime, and they pulled it off. 239 00:27:07,740 --> 00:27:14,340 So, it was, like, they were able to get into a new, a new facet of the industry, basically, because they could stay up while some of their competitors weren't, so. 240 00:27:14,920 --> 00:27:19,500 Yeah, we were, we were engaged there for a while with their reliability engineering team. 241 00:27:19,940 --> 00:27:23,400 They are just starting to explore their, their chaos journey. 242 00:27:24,340 --> 00:27:36,160 I wondered, like, what are some of the common mistakes that you see customers making or their, I don't know, that teams make in general when they're doing this type of work? 243 00:27:37,740 --> 00:27:41,560 Yeah, so the number one mistake is no monitoring, right? 244 00:27:41,760 --> 00:27:48,280 You don't actually have the ability to measure the impact of the experiment you're doing. 245 00:27:49,560 --> 00:27:52,440 The second would be no kill switch. 246 00:27:52,920 --> 00:27:55,820 You can't turn your experiment off if things go haywire. 247 00:27:57,280 --> 00:28:03,540 And that ties to the third, which would be, you know, starting in production instead of in your sandbox. 248 00:28:04,280 --> 00:28:06,060 Yeah, it sounds like timing is a big one. 249 00:28:06,240 --> 00:28:11,420 Like, like, like, do this the week before or months before Black Friday, not on Black Friday. 250 00:28:11,900 --> 00:28:13,760 The first time and build up to it. 251 00:28:13,860 --> 00:28:15,700 Start, start with your small experiments. 252 00:28:16,620 --> 00:28:18,760 Make sure you have things measured. 253 00:28:19,700 --> 00:28:28,800 And then, you know, next Black Friday, I think we'll see this retailer starting to shut down things in production. 254 00:28:29,440 --> 00:28:31,700 That's a big step to actually shut things down. 255 00:28:31,760 --> 00:28:37,280 Instead of just kind of throwing some packets away, which hopefully most systems can handle. 256 00:28:37,600 --> 00:28:46,080 When you start to shut down things, there's a pretty measurable impact if you're watching the right pieces of your metric system. 257 00:28:46,520 --> 00:28:53,320 And that gets into another issue, though, which is just, I guess, and maybe I don't know if it's a mistake that anyone makes or if it's just a misconception. 258 00:28:53,680 --> 00:28:58,060 But as with so many things, it seems like this is an ongoing journey. 259 00:28:58,580 --> 00:29:06,780 And it's not like you, like you do chaos engineering a month before Black Friday and figure you've got it, you've got it set and then never going to look at it again. 260 00:29:06,840 --> 00:29:11,680 I mean, this seems to be something we need to consistently be stressing and testing the system. 261 00:29:12,340 --> 00:29:19,740 Yeah, and the latest and greatest in chaos engineering is called lineage-driven fault injection. 262 00:29:21,020 --> 00:29:32,540 So what you do is you take, using a tool like times observability, you take all your system traces, right, how things talk to each other, and you put them into a graph. 263 00:29:33,740 --> 00:29:38,460 And you look for failure points, and then you just keep going down the list. 264 00:29:38,520 --> 00:29:41,540 What's my next biggest failure point, right? 265 00:29:41,580 --> 00:29:44,900 And as your system evolves, you keep doing that. 266 00:29:44,960 --> 00:29:46,440 You feed back into the cycle. 267 00:29:47,020 --> 00:29:58,660 What is the, out of curiosity, I mean, how is doing that in a way, can you just elaborate, I guess, on the difference between that practice and traditional cancer engineering, 268 00:29:58,720 --> 00:30:02,980 in terms of what you're choosing to, where you're choosing to inject the faults, I guess. 269 00:30:03,160 --> 00:30:05,780 Traditionally, it's not as data-driven. 270 00:30:06,600 --> 00:30:11,960 Traditionally, I would say, well, I think this is where the weak point is in my system. 271 00:30:12,600 --> 00:30:19,120 Or I would ask my team, application team, what do you think is, you know, the biggest weakness in your system? 272 00:30:20,220 --> 00:30:26,440 Whereas actually looking at a graph, right, T-Mobile, last I heard, they had 30,000 microservices. 273 00:30:28,440 --> 00:30:34,760 For someone to get their head around where the actual faults are is a lot of work. 274 00:30:36,040 --> 00:30:46,220 So if we can draw a picture and look at the picture and see how those microservices are interacting, right, because it may be a zip code service. 275 00:30:46,300 --> 00:30:49,360 We don't know who's going out there and getting zip codes. 276 00:30:49,620 --> 00:30:51,520 It's just something that's available in the catalog. 277 00:30:51,740 --> 00:30:53,020 And that may be our weak point. 278 00:30:53,520 --> 00:30:53,780 Okay. 279 00:30:53,880 --> 00:30:54,400 That makes sense. 280 00:30:54,460 --> 00:30:57,440 So that's kind of like I always think of in the big data world. 281 00:30:57,540 --> 00:31:07,080 It's like the more data and the more multidimensional it is, the more you kind of, yes, do something to run on the, like, figure out where there's an anomaly. 282 00:31:07,240 --> 00:31:16,560 And then maybe back to what it actually is, because, yeah, like you couldn't possibly, like you said, you couldn't possibly guess at the interactions between 30,000 different microservices. 283 00:31:16,720 --> 00:31:20,120 You just have to look and see what's happening and then work back from there. 284 00:31:20,180 --> 00:31:20,340 Okay. 285 00:31:21,000 --> 00:31:28,840 Well, I was just going to ask, Sean, do you think that there's any, like, up and coming trends in chaos engineering or in this type of work? 286 00:31:28,840 --> 00:31:38,700 Like, are there new practices or new things coming into play that, you know, maybe it's because there are new tools you're able to do things that you couldn't do before? 287 00:31:39,640 --> 00:31:39,840 Yeah. 288 00:31:39,860 --> 00:31:43,760 I think what's exciting to me is the community we're building, right? 289 00:31:43,860 --> 00:31:52,320 That the Chaos Mesh is a CNCF incubator project so that people can take their experiments and share them. 290 00:31:52,680 --> 00:31:53,260 That's awesome. 291 00:31:53,580 --> 00:31:58,580 I just have a final question, Sean, too, which is, you know, so that's like the future. 292 00:31:58,760 --> 00:32:09,000 The, in the present, like, what is the, the minimum, like, if you were saying, telling someone, I know, I know, I know it's different depending on the application, depending on myriad variables. 293 00:32:09,000 --> 00:32:19,820 But, like, if you were looking at, say, saying, building a reliable, a generally reliable system, I mean, are there, are there, are there things you, like, must have at this point? 294 00:32:19,880 --> 00:32:26,960 Whether it's, like, you really should be using Kubernetes or you really should be running, you know, in a cloud instances or you really should be running, like, whatever. 295 00:32:26,960 --> 00:32:34,860 Like, are there, are there, are there just things you can do even before you get to the chaos engineering part of it to ensure reliability? 296 00:32:35,840 --> 00:32:36,200 Absolutely. 297 00:32:37,140 --> 00:32:40,200 The first thing you need is a service level objective, right? 298 00:32:40,720 --> 00:32:45,160 What, what do I want my reliability to be? 299 00:32:45,380 --> 00:32:50,400 If you don't know that, you can't build anything else on top of it, right? 300 00:32:50,500 --> 00:32:53,440 For, for an iPhone launch, right? 301 00:32:53,880 --> 00:32:56,780 If I have three minutes of downtime, that's not acceptable. 302 00:32:57,200 --> 00:33:07,060 But, for the, the hunting license scenario, for another customer I worked on, well, we can be down about, you know, five hours a week, as long as it's not during hunting season. 303 00:33:07,380 --> 00:33:09,800 So, that service level objective is the key. 304 00:33:09,900 --> 00:33:11,300 And a lot of people use the nines. 305 00:33:11,780 --> 00:33:13,040 So, is it three nines? 306 00:33:13,860 --> 00:33:17,040 About 43 minutes a week that your system can be down? 307 00:33:17,260 --> 00:33:17,500 Great. 308 00:33:17,560 --> 00:33:18,520 Let's start with that. 309 00:33:18,860 --> 00:33:20,040 Then how do we measure that? 310 00:33:20,100 --> 00:33:24,220 That's where, you know, those tools become important, right? 311 00:33:24,220 --> 00:33:30,880 Something like tons of observability, vRealize operations, log insight, so I can actually know that my system is available. 312 00:33:31,320 --> 00:33:33,500 Because I can't have reliability without availability. 313 00:33:33,980 --> 00:33:47,700 Kubernetes, Kubernetes is, is a very nice to have when you get past one or two applications, because it allows you to consolidate that information into a shared work stream. 314 00:33:48,180 --> 00:33:52,000 That then tells me about my portfolio of applications. 315 00:33:52,800 --> 00:34:02,480 And Kubernetes, I mean, some platforms, they, they add additional layers of, like, you know, your application can start and can restart. 316 00:34:02,960 --> 00:34:03,400 Right? 317 00:34:03,460 --> 00:34:05,420 I mean, an application, it automates. 318 00:34:05,680 --> 00:34:09,600 I guess I'm saying, like, the platform does, does provide some degree of automation. 319 00:34:10,160 --> 00:34:17,980 Right, so, so Kubernetes will give you something like a health check. 320 00:34:18,700 --> 00:34:24,660 And then there's a couple other checks that it allows you to do out of the box, and you can extend those. 321 00:34:24,780 --> 00:34:28,760 And if the health check doesn't pass, right, restart the app. 322 00:34:29,240 --> 00:34:34,000 If the app startup doesn't pass, restart the app. 323 00:34:34,780 --> 00:34:39,860 So, obviously, those, those pieces that are built into Kubernetes add a lot of value. 324 00:34:40,860 --> 00:34:49,240 All right, I know, I think, I think we've talked about it as, like, the one-factor app in some way, that's something internally, like, that's the idea that, like, your app should be able to, should at least be able to do that. 325 00:34:49,700 --> 00:34:51,580 If it can't do that, then, like, yeah. 326 00:34:51,580 --> 00:34:51,760 Right. 327 00:34:52,260 --> 00:35:04,360 Yeah, and to reduce developer toil, you want to try and abstract those other 11 factors out from the application developer's responsibility. 328 00:35:04,920 --> 00:35:11,580 If they don't have to worry about logging, if they don't have to worry about security of the connection between other microservices, 329 00:35:12,060 --> 00:35:25,520 using something like Tanzu Service Mesh, so it can talk across clouds, then the developer can focus on providing features that can make your business grow even faster. 330 00:35:26,420 --> 00:35:26,720 All right. 331 00:35:27,200 --> 00:35:28,480 Danielle, do you have anything else? 332 00:35:28,900 --> 00:35:29,620 No, that's it. 333 00:35:29,640 --> 00:35:30,740 This has been really great, Sean. 334 00:35:30,820 --> 00:35:33,980 Thanks so much for talking to us about the cool work that you're doing. 335 00:35:34,420 --> 00:35:35,140 Yeah, my pleasure.