Welcome to another episode of Cloud and Culture. I'm Danielle Burrow from VMware. And I'm Derek Harris from VMware. If you're new, this podcast focuses on the process of application IT modernization through the lens of Tanzu Labs, previously called Pivotal Labs. Over the years, Tanzu Labs experts have helped hundreds of companies and organizations, big and small, build better software and adopt modern development practices. The tradition carries on today as Tanzu Labs experts help clients to get up to speed with cloud-native computing and the flexibility and resiliency it provides. Our guest for this episode is Sean Carey, the Minister of Chaos, who knows all about resiliency. More specifically, as his title suggests, he knows all about chaos engineering. He walks us through the evolution of chaos engineering from its modern inception at places like Netflix through today, when it's practiced by all sorts of organizations. He also gives some great advice on techniques for chaos engineering at various steps through the journey and discusses how the practice relates to other trends, such as SRE, and acceptance of failure as a tool for learning and improvement. Yeah, it's definitely an informative discussion. So I suggest if you've already made it this far, you stick around and learn from Sean why you should be paying attention to chaos engineering if you're not already doing it. Hi, Sean. Thanks so much for joining us today. Could you just give us a really quick introduction of who you are and what you do at Tanzu Labs? Hi, Danielle. Thanks for having me. My name is Sean Carey. I am the Minister of Chaos at Tanzu Labs within the Modern Application Platform Business Unit. I deliver chaos to all of my customers. Love it. When you say you're the Minister of Chaos, what does that really mean? What does that mean? Dig into that for us a little bit, Sean. What I do is I help customers understand how chaos engineering and fault injection can improve the reliability of their systems.
2:06Okay. And when we talk about that, obviously, that's more broadly known as chaos engineering for the uninitiated, right? Or for anyone who's heard of chaos engineering but hasn't really dove into it or taken the time to understand it. I mean, what are we talking about when we're talking about chaos engineering? What are some of the practices and what's the overall thing you're trying to accomplish with it? Chaos engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production. It allows us to identify weaknesses before they manifest in system-wide aberrant behaviors. Netflix coined the term, and then Google has a very similar practice called fault engineering, where they inject errors into their systems to ensure that they're handling them appropriately. Okay, great. And can you, I would love to get an example of like, what are these errors? Like, what are these things that the systems are needing to tolerate? What kinds of chaos are you subjecting people to? Well, Netflix started by shutting down machine instances to see how they affect overall performance of their content delivery network. In Google's case, what they do is inject bad data into a web request, or they can take that web request and just throw it away. And what is the, like, but I guess I think, and hopefully I think, Danielle, where you're getting at, is like, what, I mean, we're sort of testing out basically, you know, the goal here is resilience, right? The goal is to make sure, or, you know, if it's not resilience, it's something along those lines, correct? Like, to make sure, yeah, if a machine goes down, if the whole cloud region goes down, if you're getting bad requests that the application can actually stay online and functioning, correct?
4:14Correct. So, I like to think of it as an extension to test-driven development. We start with our unit tests of the actual package we're developing. Then we do an integration test with the systems we know it's interacting with. And then chaos engineering provides us a larger scale testing platform. And what does that look like for, when you talk about this, for customers, for organizations, like, what, how does one even get started going down this road? I mean, are there other tools they can use? Is it just, I mean, obviously it's not a matter of just randomly shutting down your instances and hoping for the, your production instances and hoping for the best, right? So, kind of, can you walk us through the process of even getting started with chaos engineering? Yeah, most customers getting started, what we do with them is we hold a game day. So, we ask them to think up, and when I say they, I mean the platform team generally, right? The people who are operating the underlying compute storage and networking space. And we say, what could you possibly do to your system that would impact its availability to the customer or its security to, of, of the customer data? So, we take those game days, we have people sit down in a room and start brainstorming scenarios that could have impacts. And then the second half of the day would be picking the scenarios that we think would limit the blast radius. We don't want to just shut down our Cisco routers because that's going to take down our entire system. So, we talk about blast radius and say, how can we do a small experiment and measure that impact? So, once we do that, then we identify what we need to be measuring.
6:14So, when you talk about tools, really the most important tool to have to get started with is some sort of observability tool, right? Ideally, one that can look at your applications, your network, and your compute resources to try and identify whether your experiment has actually caused reliability to decrease over the system. That would be a really great way to start. Obviously, there's a lot of other tools that we can talk about. But the chaos monkey was the first tool that Netflix put out there. And it was designed to allow developers to shut down their own systems, right? So, it's self-service chaos. This is a little more mature chaos. I'm guessing that depends based on the organization, huh? Like Netflix back in 2010, different story than Chase today, for example. Self-service chaos is a great phrase, Sean. I love that. So, yeah, I mean, speaking of Netflix, can you talk a little bit about how chaos engineering has kind of evolved since, you know, it first kind of began in these big tech companies? Yeah. So, chaos engineering has moved beyond Netflix because they were focused on the infrastructure that they were running on, which was Amazon EC2. And really, we've seen an extension into all different kinds of software and hardware, right? I think I talked about it earlier. Security is very important. What happens if we start messing with groups, permissions, denying people permissions, denying systems permissions, networking, slowing down requests, throwing out requests, changing requests. This has become pretty popular.
8:14The other big piece that the game days add a lot to is for training new people, right? So, if you have a scenario that you can reproduce on a regular basis through automation and the use of tools, as Derek talked about, we can bring our new hires into a room and say, okay, the system is broken, fix it. So, we can understand what it is they actually need to do to get it fixed and help them along that path in learning about how they are going to operate the system in the future and then potentially provide additional avenues for fault injection chaos engineering in the future. And you mentioned, you mentioned like that, you know, it started with Netflix and Netflix was running solely or at the time mostly on Amazon. What, how important is like a cloud base or how important is the platform to doing chaos engineering? Because it seems like something that wasn't really feasible in a pre-cloud or at least pre-cloud native sort of, sort of environment. Is that accurate or like any chaos engineer on, you know, way? I mean, I've been a chaos engineer long before the term was invented because I worked in a cramped data center and I used to knock cables out all the time. So, if you don't have a cloud, yeah, go pull some network cables, go pull some power cables. That's definitely an option. Seems a little less controllable. But it's a little less controllable and you cannot provide that level of self-service that we were talking about, which allows that blast radius to be minimized and gives the development teams, both platform and application, the opportunity to create new experiments without having bottlenecks. And this is kind of a tangent, but, or, but it's related. So, I'm just curious when you talk about tripping over cables and stuff, the one, it just does strike me that the one thing you often see if there's a big outage a lot of times, even though all these companies are doing chaos engineering, the post-mortem on the outage, sometimes it just comes down to a human mistake, right?
10:25Like that was not game plan for presumably. Yeah, and that's, you know, 70% of those errors are just someone typing something incorrectly, which is why we at Tanzu Labs love our pair programming, because we can see those mistakes before they make it to production. Well, it sounds like chaos engineering and fault injection is very necessary. Like when you're describing all the, all the things that you're doing and all the, you know, the, the ways you're watching the system respond and, and helping, you know, using it also to facilitate kind of training. It sounds like more than just kind of a nice to have. And I wonder, you know, what's the adoption look like out in the enterprise? Is this a common practice or is this something that folks are, you know, kind of struggling to adopt? Like where, where is kind of your average enterprise? Our average enterprise would be very, very early adoption, right? There's, there's, there's still a fear in a lot of organizations because they have very strict change control processes and change advisory boards and fun things like that to allow experimentation in production environments, which is really where you're going to get the most benefit of it. Obviously we, as I talked about blast readiness earlier, we try and start our experiments in a sandbox where if we do take something down, say on the platform side, there are no developers using that platform. And then we recommend that the platform users or application developers test in a non-production environment first, just as they've done historically with low testing and performance testing practices before they move into production. And is there a point at which it makes sense to get started because I guess the difference, like when I think about a Netflix or Google versus an enterprise application, a lot of, you know, the differences are in terms of scale and number of users and, and traffic and all these things.
12:37I mean, is there a point where, and, you know, probably architecture, is there a point where it makes sense, like it almost becomes a necessity to start doing this, whether it's in terms of adopting microservices or if you're, you know, your traffic starts picking up markedly or whatever it is where you really, really need to start doing chaos engineering. My background is in financial services and what I found was a great way to encourage people to understand where it's going to add value quickly is by measuring the effect of downtime of your most critical system, right? And if, when we looked at ours, it was about $50,000 a second. So about $3 million a minute. So it made sense to invest, right? If it's one engineer at say $100,000 a year, right? All their time spent on stopping this system from going down would be a 30 times return on investment. So that one, that one pays off. Okay. So yeah, exactly. That's kind of it. I mean, it's, it's value basically, right? I mean, if you have, if I'm an organization, I say, if this thing goes down, it's bad, but whatever. I mean, I guess, depending on the application. I would say, I would say the second piece, Derek, is, is security. Again, the average cost of a, of a system breach, according to the latest Verizon report is $3.8 million. So if you can potentially stop a breach occurring, by creating an experiment, which would minimize a configuration error in your credentials. Again, it's, it's value. The last piece I would talk about is your team, right? This reliability engineering practice that Google's put together, the, the goal of it is to reduce toil.
14:37Your operators, your developers, they don't, it doesn't add value for them to be doing the same thing over and over again. So for me, chaos engineering injects some fun into the job, go and try and break our systems for me as an engineer, as an architect, that's something I look forward to coming to work to do. And we had, we talked about SRE in a previous podcast, and that's a nice dovetail. We talked totally about, about reducing toil there too. And these two things are definitely connected, right? Yep. I mean, can you walk through, I mean, can, can you walk through, do you work, when you think of chaos engineering or when you're doing this in, in practice, is SRE and adopting some of those principles? Does that go hand in hand with, with, with what you're doing? Absolutely. You know, starting with the game days as an early adoption practice, what we can do is say, if we break this system, right? Say our payment system, will we actually find out that we've broken it before our customer calls us from a reliability engineering perspective? That's a measurement that I don't currently consume. So I would need to then go back and say, let's write some code to enter, to create a new alert that would let me know if I have this type of failure, right? So by even just having game days without actually creating any chaos, we can improve the reliability of our systems. Does that answer your question, Derek? Yeah, yeah, definitely. And you mentioned some tools previously. I'm wondering, you know, in terms of strategy and tools, like, or just looking at the larger system, like how much of your chaos engineering strategy is affected by, like, the teams, like their platform that they're using? Or maybe even the application architecture?
16:37Are there certain architecture choices that really make it easier to do this type of work? Definitely. The benefit of using a platform like Kubernetes, Tanzu Kubernetes Grid, is immense. It's the chaos mesh open source project allows you to create what are called custom resource definitions or operators in the Kubernetes world. And that gives you the ability to do that self-service chaos that's very valuable in a high-performing organization. It also allows you to extend the chaos beyond what comes out of the box, which is pretty comprehensive to things that are business domain specific. And it also allows you to give those extensions back to the community and continue to share open source software. But if you look back, I was actually just talking to another customer who has a huge web presence, right, that doesn't run in a containerized operation. And it's not quite a monolith, but there's a lot of pieces that can still be decomposed into microservices. And what we talked about with them was injecting their chaos through a proxy that would sit between a load balancer and that application internally and just take chaos-specific traffic from an internal machine that's just generating load. So you're protecting yourself from any downstream customer impact, but still able to add value on the fault injection front. And you mentioned Kubernetes. One of the things that we've talked about, I mean, for years now, I think, is this debate between whether Kubernetes is like a developer platform or an operational platform.
18:37So it's arguably some of both. But with regard to chaos engineering, do you see, I mean, where does the onus ultimately lie, I guess, in terms of, or do you see it often lie in terms of getting stuff? Is it in the application or is it, you know, on the operational side where your clients need to make improvements? Yeah, you know, having worked in Tanzu Labs on the platform team for a long time, I consider myself a software developer, even though I'm an operations guy. So the really cool thing about the chaos mesh, right, is that it can go down to the application level and create the chaos there. If you allow it, it's all role-based access control, so permissions to hit that application. Or you can go the other way and talk to your cluster API and say, hey, shut down this instance, right? Turn off this networking. So it's a platform for building platforms. And in this case, we've built a chaos platform, which allows us to impact both the operational platform and the application platform. I know I avoided your question a little bit there. Take a side. I know, I know. It's often both. It's often a gray area, but I just want to flesh out the topic. Yeah, and I think, I mean, I think you touched on this a little bit, Sean, when you were talking about kind of the ROI behind doing chaos engineering. But I'm wondering, you know, what some of the, like, objections are to doing it. And, you know, is it really just a matter of sitting down and kind of making those calculations with clients to get them to, you know, do it? Or is it, like, how do you work with folks who maybe part of the organization wants to do this and maybe other folks don't? Yeah, I think what Derek was just talking about, the separation between ops and apps.
20:40So we have a large retail customer who also, huge online presence, right? This year they had the fourth amount of traffic of any retailer on Black Friday. And so we said we'd like to try it on Black Friday in a very, very small part of the site. And then we started earlier in the summer building up to it and showing those application teams, the business owners, the ability to control our experiments when things went wrong, which is very important to have the big red button to be able to jam on and stop everything. But by building up to that, we actually ran into a production error that we were actually running the test a week before Black Friday. And we caused a default that caused downtime. Now we're estimating, because we don't have the business numbers, we caused $7 million worth of business impact. But looking at the traffic volume for Black Friday, we saved $63 million worth of downtime on Black Friday. So by showing them that we could fix the problems we had created very quickly, we were able to build up that confidence. And that's usually where people get uncomfortable is when you jump into production first and say, oh, well, we need to start shutting down AWS regions. They go, no, no, no, hold on a second. Wait, what are you doing? But if you can build up, right, that confidence level, show your maturity as an organization, the detractors tend to fade away. Yeah. And do your clients usually bring it up? Is chaos engineering something that you bring up to them or that companies would bring up to you as something they want to do?
22:43Yeah. I mean, with my title being Minister of Chaos, when I introduce myself, it gets them thinking about it. But as a topic, you know, we try and introduce it as part of our reliability engineering practice, right? When we see that they're at the maturity level where it's going to add value to their organization. And do, Sean, do some customers come seeking it, though? I mean, is that an area that, you know, say they know they want to get better at and they want to upskill their teams to be able to do it? Yeah, where we've seen more customers seeking it is actually on the application side because they're not familiar with it. And they've heard about some of our successes in other customers on the operational sides thanks to podcasts like this. I know we've heard customers like T-Mobile speak at SpringOne about their chaos engineering practice. And I wondered if there are certain industries that are more maybe ahead of the curve or more aggressively adopting these practices. Like, would you say telco is one of those industries and other others? Yeah, we're working with a customer who's, you know, building a 5G network. And just culturally, it's a new part of their business. And so what they're trying to instill in the people building that business is that failure is acceptable. So chaos engineering is something they wanted to do up front to get the people's mindset who are building this network wrapped around that, you know, failure is acceptable. And what we've seen is really it's a lot of high-volume organizations that are out in the lead, right, where 25% of your business is occurring on a single day. And you just, you can't have downtime. But realistically, right, with the complex microservices, Kubernetes, right, running 90 itself, as this complexity grows, right, if you do the math, right, the uptime gets harder and harder to keep where the business demands.
25:01And is that like, I was going to say, I mean, T-Mobile actually strikes me as an interesting company as well, because, like, you know, again, I mean, it's not like their traffic happens all at 25% of it happens in one day, for example. But like, when something happens, when you have whatever millions of consumers as users, for example, that's a different type of volume, I guess. You know, T-Mobile, I remember, I remember I did a podcast with them actually at SpringOne, like, it seems like forever ago now that we had a live SpringOne on. But it was about, you know, we talked about, like, when the CEO notices, when the CEO notices an outage, that was a pretty big indication that they needed to get on this and get on the reliability factor. So, I guess, yeah, I just, that's just wrong with me when you mentioned T-Mobile and the high volume, right? Like, obviously, they do lots of traffic a lot of times, but, you know, they just have a lot of users, period. Well, that, and the instance we worked with them initially to focus on was the iPhone launch. So, iPhone launches have become more mundane nowadays, but back in the day, right, that was, they sold a lot, a lot of phones, and they had a lot of downtime. I think the first three iPhone launch days, they had downtime. So, by the time we got our platform up and running, I think four through 11 are we on now? We haven't had any customer-facing downtime since then. Cool, yeah. And I think, yeah, the other one that comes out as a customer, that example that comes about, I don't know that it was, that you guys worked with them or that they had, and that they had worked on this was, I think it was Dick's Sporting Goods around, they had talked about, just on a similar thing to the iPhone launch. And I know in the Sporting Good world, there aren't a lot of analogs, but, like, new shoe launches can be one of those. I think it was, like, the Yeezy, this, I don't even know how old the story was, a year, year and a half ago. But, yeah, it was, like, they were able to go and do this Yeezy launch, which is a big deal if you're not a sneakerhead. That's apparently a big deal.
27:02I'm not one. But, and I was, like, yeah, I know they were talking online and tweeting about how they didn't have any downtime, and they pulled it off. So, it was, like, they were able to get into a new, a new facet of the industry, basically, because they could stay up while some of their competitors weren't, so. Yeah, we were, we were engaged there for a while with their reliability engineering team. They are just starting to explore their, their chaos journey. I wondered, like, what are some of the common mistakes that you see customers making or their, I don't know, that teams make in general when they're doing this type of work? Yeah, so the number one mistake is no monitoring, right? You don't actually have the ability to measure the impact of the experiment you're doing. The second would be no kill switch. You can't turn your experiment off if things go haywire. And that ties to the third, which would be, you know, starting in production instead of in your sandbox. Yeah, it sounds like timing is a big one. Like, like, like, do this the week before or months before Black Friday, not on Black Friday. The first time and build up to it. Start, start with your small experiments. Make sure you have things measured. And then, you know, next Black Friday, I think we'll see this retailer starting to shut down things in production. That's a big step to actually shut things down. Instead of just kind of throwing some packets away, which hopefully most systems can handle. When you start to shut down things, there's a pretty measurable impact if you're watching the right pieces of your metric system. And that gets into another issue, though, which is just, I guess, and maybe I don't know if it's a mistake that anyone makes or if it's just a misconception. But as with so many things, it seems like this is an ongoing journey. And it's not like you, like you do chaos engineering a month before Black Friday and figure you've got it, you've got it set and then never going to look at it again.
29:06I mean, this seems to be something we need to consistently be stressing and testing the system. Yeah, and the latest and greatest in chaos engineering is called lineage-driven fault injection. So what you do is you take, using a tool like times observability, you take all your system traces, right, how things talk to each other, and you put them into a graph. And you look for failure points, and then you just keep going down the list. What's my next biggest failure point, right? And as your system evolves, you keep doing that. You feed back into the cycle. What is the, out of curiosity, I mean, how is doing that in a way, can you just elaborate, I guess, on the difference between that practice and traditional cancer engineering, in terms of what you're choosing to, where you're choosing to inject the faults, I guess. Traditionally, it's not as data-driven. Traditionally, I would say, well, I think this is where the weak point is in my system. Or I would ask my team, application team, what do you think is, you know, the biggest weakness in your system? Whereas actually looking at a graph, right, T-Mobile, last I heard, they had 30,000 microservices. For someone to get their head around where the actual faults are is a lot of work. So if we can draw a picture and look at the picture and see how those microservices are interacting, right, because it may be a zip code service. We don't know who's going out there and getting zip codes. It's just something that's available in the catalog. And that may be our weak point. Okay. That makes sense. So that's kind of like I always think of in the big data world. It's like the more data and the more multidimensional it is, the more you kind of, yes, do something to run on the, like, figure out where there's an anomaly.
31:07And then maybe back to what it actually is, because, yeah, like you couldn't possibly, like you said, you couldn't possibly guess at the interactions between 30,000 different microservices. You just have to look and see what's happening and then work back from there. Okay. Well, I was just going to ask, Sean, do you think that there's any, like, up and coming trends in chaos engineering or in this type of work? Like, are there new practices or new things coming into play that, you know, maybe it's because there are new tools you're able to do things that you couldn't do before? Yeah. I think what's exciting to me is the community we're building, right? That the Chaos Mesh is a CNCF incubator project so that people can take their experiments and share them. That's awesome. I just have a final question, Sean, too, which is, you know, so that's like the future. The, in the present, like, what is the, the minimum, like, if you were saying, telling someone, I know, I know, I know it's different depending on the application, depending on myriad variables. But, like, if you were looking at, say, saying, building a reliable, a generally reliable system, I mean, are there, are there, are there things you, like, must have at this point? Whether it's, like, you really should be using Kubernetes or you really should be running, you know, in a cloud instances or you really should be running, like, whatever. Like, are there, are there, are there just things you can do even before you get to the chaos engineering part of it to ensure reliability? Absolutely. The first thing you need is a service level objective, right? What, what do I want my reliability to be? If you don't know that, you can't build anything else on top of it, right? For, for an iPhone launch, right? If I have three minutes of downtime, that's not acceptable. But, for the, the hunting license scenario, for another customer I worked on, well, we can be down about, you know, five hours a week, as long as it's not during hunting season.
33:07So, that service level objective is the key. And a lot of people use the nines. So, is it three nines? About 43 minutes a week that your system can be down? Great. Let's start with that. Then how do we measure that? That's where, you know, those tools become important, right? Something like tons of observability, vRealize operations, log insight, so I can actually know that my system is available. Because I can't have reliability without availability. Kubernetes, Kubernetes is, is a very nice to have when you get past one or two applications, because it allows you to consolidate that information into a shared work stream. That then tells me about my portfolio of applications. And Kubernetes, I mean, some platforms, they, they add additional layers of, like, you know, your application can start and can restart. Right? I mean, an application, it automates. I guess I'm saying, like, the platform does, does provide some degree of automation. Right, so, so Kubernetes will give you something like a health check. And then there's a couple other checks that it allows you to do out of the box, and you can extend those. And if the health check doesn't pass, right, restart the app. If the app startup doesn't pass, restart the app. So, obviously, those, those pieces that are built into Kubernetes add a lot of value. All right, I know, I think, I think we've talked about it as, like, the one-factor app in some way, that's something internally, like, that's the idea that, like, your app should be able to, should at least be able to do that. If it can't do that, then, like, yeah. Right. Yeah, and to reduce developer toil, you want to try and abstract those other 11 factors out from the application developer's responsibility. If they don't have to worry about logging, if they don't have to worry about security of the connection between other microservices,
35:12using something like Tanzu Service Mesh, so it can talk across clouds, then the developer can focus on providing features that can make your business grow even faster. All right. Danielle, do you have anything else? No, that's it. This has been really great, Sean. Thanks so much for talking to us about the cool work that you're doing. Yeah, my pleasure.