Why Every Team Should Embrace Chaos Engineering with Gremlin’s Tammy Butow

Episode 129 · 45:36 · SRT · archive.org · all Tanzu Talk

Our guest this time, Tammy Butow, you know, chaos engineering is one of those things that I've heard about for years, you know, the Simeon army, it sounds really cool. I can totally picture monkeys in a data center now. I never really would have thought about that before. Now I actually think that like having four-year-olds in a data center is actually way more chaotic, but that's a topic for another day. We wouldn't let that happen, would we? I mean, if you really want to see a random number generator in action, it's just like put like one of your files in front of a two-year-old and like watch them do like, there's no order to it. And so it's wonderful. It's like the best encryption you could ask for. So anyways, like, you know, I've heard about it. It sounded cool. I'm picturing these monkeys and kind of randomly shutting stuff off. And then hearing you talk last year, a light bulb kind of went off for me about like, oh, this is a lot more scientific and like the whole science process and running it as experiments, which ironically makes it feel a lot less chaotic. It's much more controlled than sort of what the name kind of lets on, but it was sort of an eye-opening moment for me. And then since then, just as it turns out, I've been hearing more and more mentions of it from the Cloud Foundry community and some of our pivotal customers and how they're mentioning like, oh yeah, we're doing chaos engineering in our PCF environment. So it just kind of prompted me to want to learn a little bit more. And in particular, what I'd love to understand is whose job is it to be doing chaos engineering? Because I can see this playing out in a couple of different ways. So I thought Tammy would be a great person to help us unpack that question.

Awesome. Yeah, sounds great. So as you mentioned, chaos engineering to me, it really is a scientific approach. We like to call it controlled chaos. So you're injecting chaos into your system, but you're doing it in a really thoughtful, planned way. And it really is a disciplined approach to actually helping you identify failures before they become outages. So when I like to think of chaos engineering, to me, it's, you know, you're going to inject failure into a system, you're going to break things on purpose. Like for example, maybe you think, well, I've set up my database infrastructure correctly. If I shut down one of my replicas, then a clone should kick off and then a new replica will be put in its place. Or for example, if I shut down my primary, then it will be replaced by a replica. And I think that it's going to take exactly this long for it to happen. But the thing is, if you don't actually inject that failure, inject that shutdown, inject that killing of that machine, then you don't ever know. And the other thing is it can change by day. So say on a Monday, when you have like heavy traffic Monday morning, if you're like a heavy traffic business, a lot of people starting work on Monday, then that could actually make your cloning process slower because you have a lot of things happening in your system. Whereas it could be much faster on say a Saturday or a Friday even. But that's why it's really good to be doing these experiments continuously. So that then makes us think, yeah, like who is the right person to be running these experiments? Because you do want it to be controlled. You want it to be scientific. You want people to be thinking through what kind of things they want to learn from doing the experiments. And to me, there are many different teams that actually should be doing chaos engineering. And I think that actually your entire company should be aware that it's happening. So when I previously worked at Dropbox, right now I'm working at Gremlin. So yeah, I've been at Gremlin for a year and a half and I joined as the ninth employee there. But before

that, I was working at Dropbox doing chaos engineering as the SRE manager for storage. So that's where I did chaos engineering. I was a site reliability engineering manager for all of Dropbox's storage, which is a lot of storage. It's for 500 million customers, more than that even. And it's all of the block data and metadata. So there, when we did chaos engineering, we did a lot of different things like DNS failover. We would do a lot of work on the MySQL machines to shut down replicas, to shut down MySQL process on the machines and make sure that everything worked correctly in terms of hardware management, because it was actually a lot of bare metal or on-prem. And to me, that makes sense. Like, you have your team that knows their system really well. So we're the storage team. We're SREs. We know what the failure could potentially look like. So we're going to plan out those experiments. We're going to run them and going to run them continuously, different times of the day, different days, every single week. I used to do at least three experiments a week and we would continuously repeat them. And then that, you can learn a lot. And I actually got a 10X reduction in incidents for that specific team by doing that sort of work, which was awesome. And I talked about that at a few conferences and it was really great. You know, it went from having high on-call load to really not having much on-call load at all. And that means you don't have to be getting paged on the weekend or in the evening. You know, you actually get to have some time to spend with your friends and your family, which is so much better. And also it reduces burnout, which is great for SREs too. And then that makes you then think, okay, if one department can get so much value out of chaos engineering, then it makes you think what other teams could potentially get value out of it. Because I think like a good thing to do actually is to focus on the teams that have the most problems first. Like that's actually a good area. Like look at the teams that have a high on-call load where it is a critical system, where there is a large number of outages or incidents. Like I always say that, like focus on the big wins and the critical systems first,

instead of looking at low hanging fruit. Because the thing is, you're going to get a lot more value out of it if you focus on your critical systems when you're doing chaos engineering. And then that generally makes people get a little bit worried because they think, well, you know, it's a critical system. I don't want to inject failure. But the thing is, you know, failure is always going to happen. Machines are always going to get rebooted. Containers are always going to randomly die. All sorts of problems are happening anyway. But if you actually inject that failure in a really controlled way during the middle of the day, while everyone's there, you can measure it, you can monitor it, you can look and learn from it. That's so much better than just being too scared about the failure. Because definitely your, you know, your replica is going to shut down. And different types of machines are going to shut down. So that's just something that we all know will happen. You'll have problems with kernels and all that sort of thing. So, you know, you mentioned that you were doing this at Dropbox as an SRE and that that's something that an SRE can, I mean, this might kind of require just maybe illuminating for us and our listeners, you know, an SRE, what's the relationship between an SRE and an app development team or a service team, right? Like if I'm running, if I'm providing the MySQL and other database services to, you know, a bunch of enterprise developers, what's the relationship between an SRE? And then we have kind of, in our world, we have like kind of the platform engineers and the folks who are providing that underlying platform and building in a lot of reliability. Cloud Foundry has a lot of that kind of built in already. But to me, there seems like there's still these other places for failure that would benefit from this approach. And the backing services and at the code level itself of the application are two that jump out for me. And so does this mean that, you know, you should have an SRE dedicated to those services?

Can they float across different services and app teams? Like what's the relationship there? Yeah. I honestly think like every team really should be doing chaos engineering and thinking about failure injection. We actually have like, you know, a lot of people talk about it from infrastructure, backend systems, databases, critical services. But then also at Gremlin, our UI engineers do chaos engineering. So we have built something called ALFI, which is application level failure injection. And what they can do is we can actually trigger for a failure to be created. And you can actually then see in staging when this failure occurs, what happens on the UI? Like what kind of error message is displayed? Do we have like graceful degradation instead of just the entire page not working? Something like that. And one of the nice examples that a lot of people would realize is usually when you're using Netflix, you have this nice like hero banner when they're trying to, you know, show you some type of new movie or TV show that they want you to watch. And it's this big hero banner across the top of the screen when you open it up. And then below there, there are little tiles with different types of TVs and shows and movies to select. But if there's something wrong with that movie that they're trying to premiere and encourage people to watch, then they'll just remove the hero banner. They can just get rid of it. But it would be so much worse if, you know, it wasn't working. And then the hero, like people tried to click on it and it didn't load or the image didn't load or there was a big X or, you know, you can imagine all the different types of things that could go wrong and how it could look bad to the user. But that's why it's really great when you have your UI engineers thinking about graceful degradation too. And this is something that, you know, it does take, it's a big cultural shift. But for me, for example, like Dropbox, it was like, all right, if we trigger this failure as the databases team and we shut down this machine, we actually would work with other teams. So we would have a Slack channel and we would have multiple teams in that channel to say,

hey, what errors do you see? Do you see any problems with your service? Can you easily reconnect to the database once this new primary comes up? Did you see any issues at all? And that was really important because you want to make sure that you squash any problems there. And you also want to make sure that when new people join a company, they know how to do on call and they know what errors look like. They know how to monitor for database issues. Like it shouldn't be a mystery. It should actually be that, you know, if you need to connect to your storage layer or your database, you should know what errors look like. You should know how to handle errors instead of just saying like, oh, my SQL's gone away. What does that mean? So chaos engineering is actually a really great learning opportunity for teams to work across the company. Really great bonding opportunity. I've just had so many amazing experiences through using chaos engineering, you know, to help new people on board for on call, especially if they've never done on call before in that team. You know, maybe they did a team transfer and this could be someone who is an SRE or someone who's a software engineer. And you can show them, hey, like this is what the failure commonly looks like. You can reproduce an old outage and make sure that that outage never happens again and show, well, we put these things in place and you can see when we trigger this failure. Last time it did this horrible thing and created this massive outage for maybe days or hours. But you can see now we actually handle it and the problem won't happen again and we're really confident about it. So that's a great way to prove that your fixes are actually working. And then another really interesting place to do chaos engineering is in your CICD pipeline or within your build team. So you can do something where you can inject chaos engineering experiments every time you do a deploy. Like, for example, Spinnaker is what's used at Netflix for their CICD pipeline. And in there you can actually click to say, I want to use chaos monkey for these these deployments.

And then chaos monkey will run for that. Yeah. So I think there's a lot of interesting things that we can do. And we're really just scratching the surface at the moment to see where chaos engineering can go. You know, it started off being all about just I'm going to reboot a random machine, which is like a really good start. I also used to do that when I was working at the National Australia Bank. So, you know, this isn't just for like tech companies and startups. I actually first did chaos engineering when I was at the National Australia Bank. And yeah, we had every single engineer needed to know how to handle failure. We would do big exercises every quarter, which are commonly called like disaster recovery. And you need to do it for compliance to keep your banking license. But that means that, you know, you get hundreds of engineers and other people across the company. You'll have like people from testing departments, project managers, like TPMs, customer support, customer success, like business owners, all coming to this big location to run the experiments and make sure that failure is handled well. And then they actually would go around and check all the different experiments off. Say, yep, you could do a region failover well. Or if you didn't, then you had to be able to prove that the next time, next quarter, you were able to do it. So you only had three months to fix it. So, yeah, I mean, and that was really good. Like that definitely helped us think in a different way. Like you're building your services knowing that region failover will happen, which is a different mindset. And it makes you think when I build something new, I'm going to build it in a way that is more reliable. But then you also know that, you know, things will go wrong and that's why you need to keep running experiments. Okay. So every team kind of needs to be doing some. Is there a group or does it make sense for any particular team to kind of take a little bit more responsibility for owning and driving the practice? And, you know, I can imagine like, you know, you mentioned Alfie and kind of injecting that at the UI layer.

But if you've got hundreds of applications, that's a lot of teams to ultimately onboard. So they're going to be in different stages of having adopted this. Yep. What's kind of a best practice that you've seen in terms of starting with one team and expanding? Like how does this practice kind of spread from one team to another? Is there kind of a third party in there that owns that maturation process? Yeah. So that's a, yeah, it's a really great question. A lot of people ask this. And I think from what I've seen in the past, what worked really well is that it is nice to have a team dedicated to owning the outcomes of the chaos engineering experiments. And at certain companies, you know, that I've worked, it was either the SRE teams or other places, they might be called production engineering or platform engineering, or it could be operations like within any space within DevOps also makes sense. But the thing there is, what's really important is, you know, if you have a large company, like say when I was at the National Australia Bank, that's like, you know, tens of thousands of people, what you need to do is really be very transparent and open. And even if you're at a company with hundreds of people or a few thousand people, you know, tell the entire company what chaos engineering is. And I think the best way to do that, if you can, is through all hands. So you can just start by having one talk where somebody comes up that says like, all right, I'm going to be the champion of chaos at our company. And I'm going to tell everyone at the company what chaos engineering is. And that's really important because you want to make sure that, for example, everyone knows that you're doing something to get good results and to make a positive impact. And that's a really handy way to do it. And then you can also have Q&A and you can open it up to everybody. If you don't do all hands, then it could be something like recording a video and explaining what chaos engineering is and how you're going to do it.

And then sending that out to the entire company. So people know this is something we're going to do. This is where it's been done before. These are the results that those companies have got from it. You could say like, for me, for a 10x reduction in incidents, reducing on call, reducing downtime. And there's a lot of great reasons for why to do it. And then just link to some further reading. But I think that's a really good place to start is the education of what chaos engineering is and why we want to do it. And then just get people a little bit more comfortable to the idea of, you know, failure is going to happen. But if we can control it and inject it proactively to make sure that our systems can handle it, then we'll be in a much better state. And everyone understands that. They know that that makes so much sense when you say it to them. And then when they see it in action, they're like, yeah, this totally makes sense. And then the next thing that you can do after just doing some education, people will get pretty excited. And then they'll want to do chaos engineering. So I think a nice way to start is to have one team get selected to do it first. And you can run something called a chaos day or alternatively, some people also call it game day. I like to just call it chaos day because we are injecting chaos. And what you do there is you just send out an email, say, hey, we're going to record this chaos day session or we're even going to stream it for the entire company to watch. We're going to inject failure into this system and then we're going to write down some action items after that to say what we need to do to fix it. But there's a lot of work that you need to do to plan a chaos day. You know, you need to whiteboard your architecture for the system that you selected. You need to make sure that you know what your halt or exit conditions are. And in Gremlin, like, we built a feature that allows you to just click a button to halt all experiments. That's really important. You also need to make sure that people understand blast radius. And that's the idea that you're going to start small and then gradually expand the blast radius for your experiments. And you also need to have a few prerequisites. So if you say, for example, pick one team that you want to do those chaos engineering experiments on, you need to also think about, all right, the prerequisites for chaos engineering are monitoring.

You need to have some good monitoring there. And chaos engineering can help you fix and improve your monitoring. But that then means that you need to engage the team that owns monitoring at your company. If you have a dedicated monitoring team, or maybe that is your operations team or your SREs or production engineers. But you need a rep from there in your chaos day. And then you also want to have someone that handles alerting. That could be the same team. Like, maybe if there's a product that you use for that. Like, for example, two products could be Datadog and PagerDuty as monitoring and alerting. And then make sure you have someone in there that can go, all right, actually, yeah, our alert thresholds aren't set correctly. We need to fix those. We haven't audited them in a long time. That might be something that you learn from this experiment. But then those action items get assigned to a different department and a different team to think more about that holistically. Well, we learn this from this one chaos day. We learn that actually, yeah, we found some gaps in our monitoring and alerting. But let's take a step back and think how we can improve monitoring and alerting across the company. And then you can enable people and empower them by saying, hey, like, if you want to run your own chaos day, this is how you can do it. Give you access to a tool, give them education, tell them what kind of experiments they could start with. So then people feel really empowered to do it and add it into their daily practices. And just knowing, like, you know, if you measure first, like, this is the most important thing that I've said to everyone is, like, you need to benchmark before you run your experiments. So you really want to know how many incidents you have, how many outages you have, how long it takes people to work on outages and incidents, you know, how many hours are people spending each week on on-call, are people having to get paid during the weekends and outside of business hours. All of those things are really important to know first. And you can even just survey people based on how effective do you think is our handling of incidents within our team and our handling of outages and get them to rate that on a scale. And then once you've got all those, that data first, then gradually each month, I think for the first three months, definitely rerun those surveys and check again to look at your data and see how you've improved.

And I found that you'll get a massive improvement within three months, definitely. For me, every single time you always do. But then, you know, you want to share those results. So start small in a company, just like the idea of doing chaos engineering, you know, contain the blast radius, start with one team and gradually expand to more teams, but show the results and show how the team benefited. Help that get that team to then share with other teams, get them to present at all hands, get them to record a video and explain what they learned from doing the chaos engineering experiments and why it was valuable for them. I think that's just a really, really good thing to do. And it will get your company just excited about making your services more reliable for everybody that's using them. Well, it's interesting. I have two small kids, so every day at my house is chaos day. But I think Domain can probably relate to that. So I totally am with you on chaos day. I do wonder, we talked a little bit before we started recording, just a little bit about Gremlin and kind of the state of the tooling around chaos engineering. I wonder if you could talk a little bit more about that. So talk a little bit about the kind of the evolution of the discipline and some of the tools that are built up around it. So I actually, I recently wrote up an article, which I shared, which is the chaos engineering principles and the history. So, yeah, that's good because I've looked through all of the history of chaos engineering research for that. And it was really interesting to just see how things have evolved and where things have come from. And I would say, like, you know, you see all the time in companies, there is a lot of things happening, a lot of different things going on, a lot of change. Hopefully, like, your company's doing well and you're getting a lot of customers. Like, that's ideal. Like, and I always like to tell people that being an SRE feels like you are trying to build and repair the rocket ship, but from the outside when you're working in a startup. Like, that's what it feels like.

It doesn't feel like you're inside. It's all comfortable and great. It feels like you're in the outside hanging on while you're trying to continuously build it and repair it. And that is just, like, to me, the best way to describe site reliability engineering. It's like, you know, there's intense and there's like, whoa. But, yeah, so that's a good place to start. But the thing is chaos engineering can help you reduce that feeling because it actually gives you a lot of control around the chaos. You can look at it. You can take a step back and go, okay, like, what are our biggest problems? Like, I like to say that. What are our top five issues that we have right now that we need to fix? Let's get the data to prove that. And then let's just try and cut them down. And I like to always mention Pareto principle. That's one of my favorite things ever. So when I was working at the National Australia Bank, we always used to solve problems using the Pareto principle or 80-20 rule. And, yeah, the 80-20 rule. So it's just like, look at your, you know, your biggest problems. And you always have this long tail. So if you can say, all right, 80% of our problems are caused by 20% of the issues that we're currently being paged for. Like, that's usually what it's actually always like when you look up Pareto principle, you'll see, like, wow, that's interesting when you're looking at data. But then in terms of the history of chaos engineering, I don't think that was ever applied. Like, that's just the way that I thought about it because of coming from a banking background. And you're always looking at so much data. And when you work in a bank, everyone's really encouraged to prove with data first what you want to do and what improvement you're going to make. You have to benchmark. Like, you won't be able to do something unless you benchmark it. You have to do a lot of really rigorous things like change records and say, these are all the different types of things that I've done to make sure that my software is reliable before I put it in production. And if you can't list those out, then they just won't let you put your code in production. So it's like, they'll actually call you on the phone and say, hey, like, you know, what you wrote down wasn't good enough for your change record. Go back and do more work to make sure that it's reliable before we let you put it into internet banking or foreign exchange trading.

I feel like everyone should work at a bank in some way. Yeah, it's actually really handy. At some point in their life. Just, I mean, I, I worked at a bank, but I mean, and I totally, I was in a, I was in the research department, but I totally know what you mean in terms of like this. The, the paper trail is like very real. Um, and you learn a lot of rigor. Everything's linked to your name. Yeah. And you know that everything's getting recorded against your name. I actually once got a fine from the, um, ombudsman because a mortgage wasn't able to be processed. And then the customer rang and complained. And then that name got associated with that fine got associated with my name. So like the, uh, CIO came and said that to me. I was like, well, okay, this is serious. And yeah, it's just an interesting thing. Very different environment, but I think I learned a lot from that. And then when we look at chaos engineering and where it began, it was more about, okay, like let's just shut down random machines on AWS. Yeah. Which like makes a lot of sense when you're doing a migration. Cause it was from back when Netflix, um, and Bruce Wong is the one who created the term chaos engineering. He then went to work at Twilio and now he works at Stitch Fix. He's a good friend of ours. And yeah, there was this big migration, which was happening, which, um, Adrian Cockroft worked on the migration. Now he's at AWS. And he actually spoke at our conference. He did the opening keynote at ChaosConf about chaos engineering and how it's changed over the years. But the idea there was they didn't want, um, Netflix engineers to, you know, think that these machines are going to be trustworthy. They were going to be up all the time. And like years and years ago, people used to say that machines were good if they had like a high number, high amount of days for uptime, which is like just really bad idea. You know, it's better actually if they constantly cycle. So then they're healthy. Like I always used to think, I like to think of it as a healthy fleet, which means like they've been washed or like looked after when you think about it, like cattle. Um, you need to do hardware, like upgrades and you need to do security patches.

So having really high uptime actually isn't very good. Um, so that was just changing that mindset and saying, okay, everyone just get ready. Like these machines, they're not ours. We don't control them. They're owned by AWS. They're probably going to go down. They're probably going to bounce every now and again. We need to make sure that we handle that. Um, and so it was just killing random machines. And then after that, um, at the same time, or like even a little bit before Colton and Forney were working at Amazon and they were building an internal tool to do chaos engineering. And they were looking at actually injecting failure on, um, Amazon because they knew that all these people are going to come and start using Amazon. This is like many, many years ago now. And they wanted to make sure that it was reliable and they built this tool and gave it to everybody across Amazon. So, and people still use that tool internally now. Um, and yeah, I think that's pretty cool that they did that. They had a lot of, um, foresight to think this is what people will need and we need to give them like a really nice UI, um, an agent that can run on the machines. We want everyone to be able to quickly adopt it and use it. Um, and then after that, then Colton went to work at Netflix and he built Fit, which is a failure injection platform. And then that was more about like taking the learnings from what he'd done in the past and then applying those to Netflix as well. And the idea there is, um, the idea is, you know, okay, I want to inject failure, but I want to make it very narrow in scope, kind of like heart surgery, very precise. Um, and the idea is you could say, I want to inject failure on just my PlayStation 4 at my house, just for my user, my user ID, instead of shutting down, you know, lots of different machines or something like that. You're just injecting failure in a very precise way. Um, and then, yeah, at the same time I was working at Dropbox doing chaos engineering on databases, which was a really new field. No one had done that before because they're all a bit scared to inject failure on a database.

But, you know, I, I mean, I just have this idea that I've been working on databases for a long time. Um, and when you have, you know, that they're going to fail all the time, that's just what happens. Like database machines will shut down and you need to have proper replication. You need to have good backups. You need to test your backups. You need to have cloning in place. Um, that's all really important, but this is a thing that most people were too scared to touch. And then, yeah, I went and spoke about it at some conferences, what we were doing and, and the results that we'd got and people were like, wow, that's actually really cool. Um, so a lot more people started to do that as well with MySQL, which I think is great. And Netflix had done some chaos engineering with Cassandra on AWS, which is also good. But yeah, there's a lot of things that have happened over the years where it's like, gradually people have felt more confident and tried to see, okay, can I use chaos engineering to fix this issue that I have? And then they've got great results from it and done it. Um, and then when I first saw the gremlin tooling, that was after I'd been at Dropbox for two and a half years and it just looked awesome to me. Like I had no idea that something could be so great. When I first did try the demo, I was like, wow, this is great. Cause it had 11 attacks built in and it's got a UI, a CLI and an API. And then it also has Alfie, which is the application level failure injection. And some of those attacks are packet loss, packet corruption, latency injection, um, time travel, like changing the clock time, lots of different things. And there's a lot of experiments there that, you know, I'd never even thought of and I'd never heard of. No one had ever spoken about them except for Colton and Forney. Um, so yeah, I just thought, wow, they're really ahead of where everybody is. And it's been awesome to be able to, uh, use gremlin and learn more about how we can actually identify failure. And we're identifying failure every single day within our own infrastructure. We're able to use gremlin to make it more robust and resilient before we say move something into production.

We do a lot of work with gremlin on staging first. And that's also a common question. People always say, is it okay to do chaos engineering on staging or in my local development environment? I'm like, yeah, totally. Like you can even start by doing it on your own laptop, using Docker for Mac, uh, start doing some chaos engineering there by learning about what it's like. Cause I think it does have, um, you know, you need to learn what it is like, how it works and gradually ramp up your chaos engineering practice. Because, you know, I always tell everyone, don't just go in tomorrow and be like, yeah, let's shut down the data center. Like that's the worst way to do it. You know, you want to start slow, um, and just contain that blast radius for your own experiments. Yeah. Yeah. So that, I mean, it's kind of interesting, your explanation of the really targeted failure injection versus sort of the, maybe the long-term state, which I think you described was sort of the Netflix model of, you know, not wanting the engineers to treat infrastructure as reliable, which is really just more of a mindset shift. But, you know, these kind of chaos engineering practices reinforce that mindset because, you know, they, it could affect them. Right. So it's on the one hand, you don't want to start with, Hey, we're just going to shut something down and I hope you're ready. But you do kind of want to get to that point. It sounds like, I mean, everyone should start small, teams should start small, but is the end goal to be able to have, Hey, this whole company is now operating this way where if we say, we're going to, we're going to attempt a really big failure, um, that could have a really large blast radius. So like, hopefully everyone's ready for that, but yeah, you definitely want to get there. Exactly. Like you said, the thing there is like, you know, to think of, um, all right, I start small with my blast radius and then what does it look like when it's really big? Like a really big example is your region failover, um, or data center failover. So that's always really good, right? If you have, um, an AWS region on the East coast and on the West coast, and then you feel confident enough that you can shut down an entire region.

And that, you know, that's happened like S3 on, um, had a massive outage. A lot of the internet was down, um, and having problems because S3 wasn't working in a region. And that was a few years ago. I was on call for that while I was at Dropbox. I was the, um, incident manager on call for all of Dropbox at the time. And yeah, it was a big outage. It was interesting. It went for five hours. It was a really long time as well, but that's the thing, you know, this failure is going to happen. So you definitely want to be prepared for it. You want to be able to do region failover. Um, but the thing is like most AWS services are not built with region failover out of the box. So you're going to have to make sure that you make that happen yourself. That's a big thing. Yeah. Um, no, we definitely see that a lot with, you know, if what you stitch together versus saying, and with Cloud Foundry, you get a lot of that automated, but it's still something that you want to test. Um, yes, exactly. Um, so another question that I have is kind of, you know, thinking about how much harder is it to adopt and introduce these practices with like a legacy monolith versus something that's already say 12 factor kind of designed, if you will, or, uh, you know, as a microservices architecture. Like, what does that journey look like? Do folks have to kind of refactor before they can effectively do this? Or is it, does it just mean that they're going to have to go about it in a different way? Yeah. Uh, I love that question too, because a lot of people ask it all the time. Um, and the thing is like, I've done chaos engineering on monoliths and also on microservices. So both. And, um, I think actually chaos engineering is really good when you're doing any type of migration. And usually every single company is having a migration occurring at any time. That's like always happening. There's something that you're doing. Um, maybe you're migrating from monolith to microservices.

You're migrating to a different type of storage layer or database, or you're upgrading your API or something like that. Or maybe you're doing a database migration or, you know, deprecating some type of system, moving everything one place to another. So this is a big thing that customers can use chaos engineering for. Um, and obviously that's like a really scary thing when you're doing a big migration, because you need to make sure that you think about like data consistency. You don't lose data. Um, I know, you know, my space just recently had a thing where they lost like 50 million songs, um, during a data migration. So yeah, that's the thing. Like you can do chaos engineering to help you prevent data loss. Like that's a big reason that I do chaos engineering. Um, because you want to make sure that you're injecting that failure to see like, if this happens, would we lose data? Um, but you do it not in production first, right? You need to, and you really think through it and document it, make it a very, very, um, controlled experiment. Cause obviously that's quite scary. If you like lose all of your data for your customers or you have data consistency issues. Um, but I think that would be really good for people, you know, with cloud foundry when you're thinking, okay, this is how I expect my system to operate. Um, this is where I want it to be, but then you actually use chaos engineering to test that it is going to work for you. That's just like the best way to do it. Yeah. I mean, I think sort of maybe a little bit of where the question comes from for me is that often I, I hear from folks where they, when they're dealing with a legacy monolith, one of the challenges is that they're so behind on doing anything with that. Because it's just, it's just really hard for them to do the right thing in any way because they're, it's just so brittle. And so they're just so they're drowning in updates. Right. Yeah. And so working through those updates, kind of thinking about like, you have to kind of bite off in some ways a lot at the same time, but then you end up in a place where it becomes easier to not only make changes for the functionality, but also to make the kinds of changes where it's like, we're going to test these changes in new ways.

And you have your, your CI pipeline set up so that you're running all kinds of tests, chaos and otherwise. Yep. Yeah. And I think a lot of that is really like a mindset shift to often people feel overwhelmed at the amount of work that you need to do. But then the number one thing that you have to be really good at is prioritization. So that's where Pareto principle comes in again and you go, all right, what are the top five things that we need to do? Let's pick off that top number one thing. Let's make sure that we have the right engineers in the room. We have enough people. We've talked about headcount. We've talked about budget. We've talked about how long this is going to take. You then take that, I think, to your VP of engineering or your CTO and you present that and say, these are the top problems that we want to solve. This is how we're going to solve them. This is how many people we need to solve them. And to me, that's about like stepping up and being really brave and courageous, which I think is something we need to do a lot more in engineering. Because there's always so many things to do, but if you can prioritize and you can be brave and you can take that to your leadership team and say, look, like, I know I can do this. I believe in myself. I'm going to be able to get us a massive reduction in problems. I'm going to make sure that this outage never happens again. I'm going to reduce that on call. This is my goal. And then you actually get your entire team behind that to work together to make it happen. Like, you're going to feel awesome at the end of it. And I think we just have to believe in ourselves and our skills a lot more that we can actually do it. That's something that I've always been like myself, just personally, as a personality trait. You know, I used to be a sponsored skateboarder and I love practicing and getting really good at something and just like being able to say, okay, today I'm going to ollie down two stairs, which is like jump down two stairs. Next week, I'm going to go down five stairs. And, you know, you practice and you fall over and sometimes it hurts, but you have to get back up and you have to keep trying. And yeah, to me, that's like a cultural shift. But a lot of SREs, you know, as a speciality, it's, you'll notice that SREs and production engineers like really care deeply about reliability and don't ever want to lose data and understand that that's like so important.

Because if you lose data, then you don't have a company anymore. Like if you've lost your customers data and if you don't have a product that's up and available, well, then your customers can't use your product and they're going to lose trust in it. So, yeah, I think we're always really pushed by that, that there's like deep empathy for customers and just knowing that you have to step up and just make it happen. Great. Okay. Well, we're getting close on time. So why don't you tell us a little bit more about Gremlin? I think our listeners love you a little bit more about the company and what you do and how, maybe how they can get in touch and potentially take a look at some of your, at your approach. Yeah, sure. So we recently launched Gremlin free. So you can actually now use Gremlin totally for free, which is cool. So if you go to gremlin.com, you can use a few of our attacks, which are shutdown and CPU. And it works across all clouds. So it's cloud agnostic. And you can also attack instances or hosts, or you can attack containers or Kubernetes pods. So that's really cool. I think that's great. That only happened just recently now. And it's been exciting to see everybody start to use chaos and engineering a lot more across many different companies. And I've seen, been seeing people reach out to me on Twitter, just sharing what they're doing, which is really cool. If you look up the hashtag free the gremlins, you'll see some cool tweets from people where they've shared like, these are the experiments that I'm running at my company. And here's what I've learned so far. So yeah, I definitely encourage you to try out Gremlin because you can. And yeah, it's totally free now, which is awesome. And then apart from that, like, you know, the biggest thing that I'm doing right now this week, you know, we're in a startup. I started, I was the ninth employee. Now we're over 50 people. But this week I'm actually running a big game day internally or a chaos day on our own infrastructure. So yeah, we're doing that this Friday and really looking forward to that. And we're actually inviting the whole company to come and watch and learn.

So people from every single department are invited to come along and understand how chaos and engineering works and why it's important and what we're learning from it. And we're going to be focusing on improving our monitoring and alerting to make sure that everything's set correctly and that we feel really good about it. So yeah, that's a big thing that I'm doing this week. Is there like, are there snacks, you know, like if you're in the audience, like is there popcorn? I'm just picturing like, it's like a shot. Yes, I think it should be like that. You got a blanket. You're like, oh my God, oh my God, is that going to fail? And then it's like, oh, look, was there, oh no, there wasn't an alert. There wasn't an alert. I didn't see that. Like that's what it feels like. That's what it feels like. Even when you're doing it on staging or locally, like in your own development environment, I think it's good to have a theme and make it something that is really memorable. And everyone's excited to come along too. And they feel like they're going to learn something from it. So yeah, when I, you know, you know, because we're gremlins, so we can use like the gremlin theme of having all these little gremlins going through and, you know, have some green cupcakes. We've definitely done that before. But like, I mean, yeah, you need to think of what sort of incentives you're going to have to get everyone to show up. That's for sure. Yeah. But like, then you don't, you, I don't know if you ever saw the Seinfeld episode where Kramer goes to watch like some kind of surgery. You mentioned heart surgery before. So this is, of course, where my brain immediately goes. It's not like a real surgery, but like a surgery on Seinfeld. And he's like eating junior mints and he drops a junior mint, like into the patient, into like the open cavity or something. So you definitely don't want that. You don't need to have some guards. The two-way mirrors or something so that there's no, you know, like junior mints getting dropped in. Yeah, it's a good way, good way to think about it though. Cause you know, when they first did chaos engineering at AWS, like Jesse Robbins used to do this thing where he'd actually go in the data center and just pull out random cables. Like that was how they first did it before they built anything.

And he's the person that coined the term game day. So yeah, that's where it started, where it was like, you know, wild and all sorts of different things were happening and nobody knew what he was doing in the data center. But yeah, mostly now we do it remotely. So they're on like a zoom. That's a lot safer. Yeah. Point cables. I like it. The four-year-olds could definitely help out with that. I don't know why TV references are, we've had one so far. I'm thinking of the office episode when Dwight pulls the fire alarm and everybody thinks it's a real fire. I'm thinking. That was chaos. It resulted in chaos. Yeah, that's another big thing. Like, you know, there was, there was that, um, the alert in Hawaii where they said there was like a massive issue there, a natural disaster. But then it actually wasn't like, that's a form of chaos. You know, you want to make sure that you're protected against the wrong alarms going off because that can be really bad. Yeah. I mean, it's so interesting to me. I think a lot about chaos engineering through movies as well. There's been a lot of really interesting examples and I think it's good for us to go. Okay, yeah, like failure happens and we just need to be ready for it. Yeah. It's actually like less dramatic. Um, but you know, then we can just go back to our sitcom lives instead of the drama lives. You know, as much as you want to give the Academy Awards to the dramas, you don't actually want to be living that life. Exactly. Like, so true. Like that, you know, I think as someone who's been on call for many, many years, like my favorite type of weekend is one where I don't get paged. Like that is the dream. Um, like one where I can actually, you know, go out, have fun with friends, not having to be getting paged all the time. That's the best. Get your weekends back. Exactly. It was a pleasure talking to you, um, and learning more and kind of just getting, getting a clear picture of sort of where and when and how and who should be doing chaos engineering.

So, uh, thank you for sharing your, your wisdom with us, Tammy, and, uh, look forward to talking to you again. Thanks so much. That was really fun to talk with you both. Thanks for having me.

Tanzu Talk · 🍕 Get more pizza.