(Almost) everything you need to know about SRE

Episode 195 · Fri, 18 Dec 2020 15:00:00 +0100 · 34:30 · SRT · archive.org · all Tanzu Talk

Welcome, listeners, to the Cloud and Culture Podcast. I am Derek Harris from VMware. And I am Danielle Burrow from VMware. If this is your first time listening, here's the quick background. This podcast focuses on VMware Pivotal Labs experts and clients who share their experiences, tips, best practices, and opinions on app modernization and digital transformation. For a more thorough introduction, listen to a previous episode. Yes. And this week, we are joined by Hannah Foxwell from the Pivotal Labs team. And Hannah explains all about SRE, which some people might know as Site Reliability Engineering. And Hannah and I discuss some of the principles behind SRE. We discuss how SRE helps with the retention of employees by making their lives easier and reducing the toil they have to put up with, and just generally making sure your applications are up and running and you're not wasting a lot of cycles over engineering for high reliability. So very insightful discussion. And you will learn everything you ever needed to know about SRE now. All right, Hannah. So thanks for being on. Can you start with just telling listeners a little bit about yourself and your role at VMware? Yeah, absolutely. So I'm Hannah. I am the Director for Platform Services here at VMware Tanzu. So essentially, my team is the field engineering team. We exist to support the engineers within our customers who are picking up Kubernetes and these technologies for the first time. Our aim really is to build wildly successful platform teams with them. All right. And so, I mean, that's a broad charter, I think. You know, one thing I think that we want to talk about specifically this week, as it relates to that, obviously, we can touch on the topics as they're connected. But it's SRE, systems, or I guess it's site reliability engineering. Yes, site reliability engineering. I was going to say system, but it is site. So can you just walk, give listeners a quick walkthrough, like, you know, what is SRE in a nutshell, for anyone who's unfamiliar? So I always sort of give the original definition of SRE, and then I

can give you my kind of take on it. So Ben Treanor, who was the founder of Google's SRE team, said that SRE is what happens when a software engineer is tasked with what used to be called operations. And I think, you know, when O'Reilly published the SRE book in 2016, like, SRE sort of launched into our consciousness as a tech community. And we picked up many of these practices, but it's evolved a lot since then. And there's a huge community around this. It's not just happening at Google, it's happening at tech companies all over the world. And I think for me, SRE is about these tools and practices that can help teams reason about, prioritize, and deliver the reliability that their users need. It's not necessarily about massive scale, highly distributed systems, or always achieving five nines of availability. It's about delivering exactly the right amount of reliability for your users, and balancing the benefit of that reliability about the cost of delivering it. Yeah. So I think that you hit upon that. It's like, it's not just, it's a set of principles, basically, right? It's not, like you said, it's not kind of, it doesn't have to be widely distributed systems. You don't have to be running Kubernetes in order to take advantage of this. It's really kind of almost a mindset shift in a way. Is that, is that fair? Yeah, absolutely. Absolutely. So, you know, to deliver any kind of software product, you know, that reliability is fundamental. But I think what SRE gives us is it gives us a really powerful language to talk about failure. It says, yes, reliability is fundamental, but failure is to be expected. It's normal. And, you know, we not only need to invest in the prevention of it, but we need the processes and practices to be prepared for it when it happens. Right. And so let's walk through some of the concepts around SRE, because again, like, I think the people are familiar with SLA always becomes like, we always, it seems like whenever any discussion of SRE, you have to define like how S, how an SLA or a service level agreement ties into this,

but it's not that, right? I mean, we have service level, I mean, SRE talks about service level indicators and objectives. You can just kind of walk through the general jargon or, I guess, the terminology. Yeah, absolutely. And I think this is like, for me, this is one of the absolutely sort of most valuable tools in the SRE toolkit. It's this ability to set your SLIs, your service level indicators. And, you know, these are to be the sort of unambiguous metrics by which you measure your reliability. This is something that you can automate the measurement of. It's not subjective and really it needs to be sort of observed from outside your system. You need to get the user's real world view of the reliability and the availability. And so you set your SLIs, that's the what, like, what are we going to measure? What can we use to get a good view of our system's performance? And then you set an SLO, which is the target level that you want to achieve on that metric. And usually it's expressed in a number of nines. For example, three nines would be 99.9% is the amount of time that you want to be achieving your SLI. The other piece of jargon that comes along with this, which I think is really important, is the inverse of the SLO. So if you say that your SLO is three nines, then you have a 0.1% acceptable failure, and that is the error budget. So you set your SLO to be your objective in terms of reliability, and you measure that unambiguously with an SLI, a service level indicator. But what you also do is you measure the amount of times that you're not achieving that. You measure the amount of times you're failing, and you keep that within a certain acceptable budget. And if you start to look like you're exceeding that budget, you can actually change your behaviors. You can make decisions based on that, and based on the user's experience of reliability about whether or not you

want to introduce more risk into your system at that time. And the one thing that always struck me when I've talked to folks about this is the idea that if if you're overshooting, so if you're never even coming close to hitting your error budget, it, right? Like maybe you've built too much reliability into the system. Is that Yeah, it seems counterintuitive, but it is a matter of my understanding is it's a matter of, like, how you want to spend your time. And if you're spending too much energy to keep at five nines, and really, like, really, it's not exactly really, it's not needed. And I do I do think that, you know, this, you know, some of these behaviors, they're driven by some of the some of the more legacy practices that we had in operations, like every outage is a disaster, there's nothing such thing as an acceptable outage, you know, and when you get into that mindset that every service interruption and every outage is, is a disaster, then you start to, you start to really over engineer your solutions to, to try and achieve the impossible, which is 100%. And, and I think these, these are tendencies that exist within all engineering teams, you know, preparing for the absolute worst case scenario over engineering solutions. And if you're achieving a very high level of availability, that really your users don't need, it means that you have, you have probably invested too much in it, whether that be through engineering time, whether that be through redundant resources, maybe you've built a lot of resilience into your system that really wasn't needed. Maybe it was through automation that you then need to maintain over the long term, and, and creates toil for your team. There are lots of ways that this excessive amounts of reliability actually cost you, not just in terms of, you know, the fact that you actually could be potentially shipping features faster to your users and taking a few more risks in the application software development lifecycle. So yeah, I think that's definitely a matter of, of balance. And the other thing too, before we,

because I want to get into how, how we've seen customers using it and how you've worked with organizations to, to adopt it. But I do think, right. The one thing I've also heard, I've talked, you know, speaking with folks from Google or whomever about it over the years is the other thing is, yes, like getting in terms of where you're spending your money thinking about, or your time and energy and toil is not just like over engineering, but also figuring out what you actually need to be tracking. So it used to be, we would track, you might be, you might've had 700 alerts on, right. Oh my God. But like six of them are the ones that actually matter. Right. I think that was the other big thing. And that just from the get-go that I think was a, for me, it always stands out when we talk with us, I read and get people familiar is like, it's about cutting down ideally or paring down what you're, what you're focusing on. Yeah, absolutely. And like the powerful thing about SLIs is it's really, it's really about, you know, what's important to my users. You know, we can monitor CPU and we can monitor error rates and things like that, but they're very difficult metrics to translate into real business impact or real user impact. And so if you, if you spend a bit more effort, you monitor less metrics, but you monitor the metrics that matter, you know, you don't end up with this scenario that we see so often, which is like alert fatigue. Yeah. You know, your page is going off every night. Yeah. You're not getting the sleep that you need to be. You're getting interrupted in your like evenings with your family. And for no reason, there was no service interruption and your users didn't care. I think of that. I used to have a ring doorbell and it would go off when like the wind blew. And then, so you'd get these, the chime would go off all the time and eventually just ignore it. So something must happen. Yeah. Oh my God. I might get set off by spiders. It's the most, it's the most terrifying thing. Yeah. That's, that's my, that's my everyday example of alert fatigue. No, no. It's the perfect example though, isn't it? It's the perfect example. It's like, yeah, I want to know when someone's actually at the door that, that I care about. So, so I wanted to ask for, in terms of like customers and, and, you know, pivotal labs clients,

I'm curious to hear kind of how you've worked with them or maybe, I don't know, this is probably, this is probably too broad to start with, but so feel free to jump in where you want. But I mean, I'm curious about like, you know, where they, I'm curious about success stories, obviously, but also like where they've run into issues and, and kind of, you know, the real world where the rubber hits the road of doing this. Cause the terminology is one thing and the concepts are one thing, but then doing it obviously, you know, probably a whole other ball game. So. Yeah. I mean, I could, I could talk about the, the challenges to adopting this sort of thing all day. I'll talk about, I'll talk about a couple of things that I consider to be success stories though. So I think one thing about defining user centric SLIs that, that really changes the perception of, of systems is, is, is that actually in a lot of organizations, there are no unambiguous metrics for delivering availability. And when I was talking to, to some customers who had a very sort of traditional mindset, it turned out that their view of availability was really subjective. And it was really about, you know, the service manager was like negotiating the number of minutes downtime that they would record against a certain incident. And, you know, some incidents weren't even recorded. Like some of them were like, Oh, that was outside of your control. So that doesn't count, but it still impacted users. So this, this is, this is what troubled me about it. So you, they were achieving very, very, very high levels of availability on the face of it. But behind the scenes, there was this person to person negotiation of what counted as a service interruption. It wasn't about the user's perception. It was about the relationships and whose fault it was and which system went down. And I think, you know, that just, that just prevents you from having good conversations about where you need to invest in better resiliency and better practices. But, but yeah, so I introduced, I introduced this one team,

like the place where this anecdote came from to the concept of like SLIs and SLOs. And they started measuring the availability of that one system that I was working with them on unambiguously for the first time. And of course, like it wasn't what they thought it would be. It didn't, it didn't compare at all to all these other systems where they were achieving very high levels of availability, but of course they weren't counting every outage as being real or valid. And so I think just introducing these different ways of, of talking about it in, in small pockets within an organization can help set a really good example of, you know, like here are our stats and we can absolutely stand behind them. We know that we've delivered this. And we also like, we know when we've overachieved and we know when we've underachieved and when we've underachieved, we've invested in like these three things to improve our resiliency and we got better results and we can show you that we got better results because the data is unambiguous. It's measured from a user point of view. Interesting. So, so it almost sounds like it's like, like it's might be a cliche, but it's a cultural thing to some degree. Oh, it is. Yeah. Okay. All right. Because the technological fix might be, like you said, changing what you're measuring and setting up, you know, setting up reliability in areas where it actually matter, you know, that matter, it affects the user experience. But in the end of the day, it's like, if the organization cannot come around to changing its mindset around placing blame or, you know, assigning blame, I should say, or around. Oh yeah. Well, I was going to say, like the challenges that I see are exactly those things. It's the, it's, it's some of the kind of fear, fear of production, fear of outages, fear of instability. And, and, you know, also the, also that sort of blame culture where, you know, you will, you will sort of identify a root cause or a team that did wrong or a person that did wrong. And, you know, those, those incidences, if they're managed like that can be really

traumatic, they can make you feel like you don't want to talk about something that went wrong. And of course, if you don't talk about it, if you hide it, you can't learn from it and improve it. Yeah. Actually, I mean, yeah, the, the blame thing is I just heard T-Mobile talking about this at SpringOne a couple of months ago, and they're talking about how they handle communication when there's an outage. And there was the big thing was not trying not to assign blame, like fixing the problem. And then, you know, if there needs to be some sort of responsibility, whatever, like you deal with that later. But the first thing was just fixing the problem. But I think, yeah, that gets to, yeah, just, just generally speaking, this idea of always needing to assign blame. And it's just a cultural mind shift that, that gets inhibits a lot of, I think cloud native, if I consider SRE to be a cloud native practice in a matter of speaking, I think it. Yeah. I've actually like talking, talking about blame. I've had, I've had somewhat of the opposite scenario as well. So I was talking to one customer about how they might, they might adopt some of these practices in their team. And they, they had, they had quite a legacy, fragile system that they were dependent on. A lot of their services were dependent on like the availability and reliability of this legacy system. And they felt almost like almost powerless in this situation that they couldn't improve their own performance and their own availability, because it was always, it was always degraded by this other system. And I said, well, the first thing you can do is so you could still start measuring it because then you have a business case to invest in this other system. You know, reliability is a team sport in this case. Like you don't, you're not delivering it in isolation. So yeah, I mean, a conversation that maybe started with a bit of blame, like, oh, we can't, we're helpless. We can't do anything about it. It's their fault. Actually, you know, I think it's, it's still a good idea to measure the actual availability of your system because it's not possible sometimes to deliver that reliability that your users need in isolation. And as long as you keep your users' needs in mind, you should be able to make improvements and make sure the

investment goes to the right places. Yeah. I like that. Reliability is a team sport. That's like, that's a T-shirt. Yes. Do you have, do you have a, I mean, the other thing I'm always curious about is whether organizations can, SRE is something you can piecemeal and pick and choose the pieces that make sense. Cause I've heard, I've heard some folks talk about it, but then they just, you know, they said, but we're not doing this, this, this. And I found myself going, are you really doing it then? So, so can you pick and choose what works or do you kind of have to buy in? Whole hog and, and really embrace it. I mean, you're not going to, you're not going to learn all of this new stuff and implement everything overnight. So I think as long as you have an intent to start and continuously improve, then you're doing all right. You know, I can talk about teams who don't use SLIs and SLOs, but they do do blameless postmortems on, on their incidents. They do create that blameless space where not every outage is a disaster and it's an opportunity to learn and improve. So that, that in isolation delivers a, a, an amount of value. And also, you know, when we talk about eliminating toil and using automation to do that, to, to, to build a software solution to what would be like a manual human repetitive task, you know, that's again, something that has value as a standalone practice. You know, you can reproduce more consistent environments using infrastructure as code and configuration management systems. You know, you can, you can rebuild your pre-production environments overnight. If you script it in the right way, these things improve the consistency and reliability of, of, of those things in isolation, but you're not necessarily going to get all of the benefits that all of the other SRE practices bring you. So, so, so what are the like reliability, increased reliability aside, which obviously

is a big deal. I mean, cause you mentioned toil a few times. So I'm, I'm, and, and I think reducing toil is, is, is one of the big, the, the big benefits. Okay. So can you talk about, I mean, let's say reliability aside, what are the, what are the benefits that, that clients are seeing as as they're, as they're embracing SRE? Wow. Yeah. Let's, let, let's talk about toil, because this is, this is one of the, one of the reasons that I took an interest in SRE to start with, because I was, I was, I was working on a platform team of a large e-commerce retailer in the UK. And we got ourselves into a situation where, you know, there was so much toil that there was no longer time in the day to automate the things that we knew we needed to. There was so much manual work to keep the lights on, as you might say, that, that even though we knew the best practices, and even though we wanted to do the right thing, we didn't have enough time. And so I think, I think the, the, the sort of Google kind of standard around this is to keep, keep an eye on your tool, toil and cap it at 50%. Like if you cap it and it starts to get towards that 50%, like we're spending half our time on sort of manual work or work that has no enduring value, then we're going to get ourselves into a situation where we no longer have time to do it. And all we do is add more humans to do more of this manual work. As a, as a fantastic quote in, in the SRE book, which is, which says that if we have to starve humans to do the work, we're feeding the machines with the blood, sweat and tears of human beings, which I think is a fantastic quote. And it sort of some, sums up my experience of, of what not to do when I was on the other side and when I was managing a platform team myself. Yeah. And it just brings to, brings to mind every, like disgruntled sysadmin.

Absolutely. Yeah. Like, and you know, it's, it's such a stereotype within so many organizations that, you know, the, like the ops team are just overwhelmed with keeping the lights on that they don't have any time for continuous improvement, but it's so true. And that, and measuring and managing your toil is, is one way to keep, keep that in check and make sure you don't go down that slippery slope. Yeah. Well, and I assume, and I'd love to hear if you've seen this in action as well, but like as, as the world becomes more digital and as every, you know, software becomes more important within companies and, and more traditional companies are competing maybe with tech companies for talent, even like, like if, if, if you're going into a place where that's going to be your day to day, you know, and you're a talented person, you might not stay long. No. So I think there's a degree like, yeah, like, so like life, your work life balance or your work experience is actually going to matter for the company in terms of retention or in terms of attracting talent. And that's, that's not a small deal when you're, when you're competing with companies like Google or Facebook who will. No, absolutely not. And, you know, like, you know, every, every sort of, every software engineering team has something to offer, but you have to think, you know, there's, there's a lot of companies who are paying, paying very highly in this space at the moment, especially now that it's a global workforce and we're all working in a very distributed way. You know, what are you going to do to make sure you keep the talent, you know, and if you can't compete on maybe salaries, then you have to, you have to offer something else. You have to offer that, you know, benefits in terms of health and, you know, using, using best practices and giving the engineers the tools they need to do, to do their job and to do more interesting problem solving and personal development than if they're just holding the pager, keeping the lights on. Right. And I say compete with Google and Facebook, but realistic, I mean, the more, the more realistic thing is probably competing with other large companies in your own city or in your

geography or whatever. It's not, it's not just the big companies anymore. And that's kind of the important thing. Yeah. Absolutely. Yes. Everyone. You're right. Yeah. Yeah. Are there, are there technological solutions that come into play? I mean, I know like, again, SRE is not, you know, part and parcel a must have for digital transformation. You don't need to be using Kubernetes. You don't need to, like, like these, all these buzzwords do not necessarily, or buzzwords is kind of pejorative, but all these, all these things do not necessarily overlap with one another, but are there, do you see certain technologies or certain practices that, or, or, you know, or processes or tools or whatever that actually do tend to help out a lot when, when trying to adopt us or re-principles? Yeah. I think, you know, I've, the type of work that I do with my team is very much about building, building platforms. And, you know, we, we tend to, we, we tend to skip ahead somewhat because of our years of experience to, to automation, because we know that if you don't automate from, from day one, you're going to have some debts that you need to pay off in the future to keep on top of your toil. So we almost shortcut, shortcut that process of introducing automation on day one. But what we also like to do is we also like to measure our success in, in building these platforms unambiguously. And so that requires a tool, some tools in place to, to measure your service level indicators. And, you know, a lot of people think that these things are only valid in production, but they're valid in pre-production as well. You're not going to be aiming for five nines availability in your pre-production environment, but there is a cost associated with them going down. And so, you know, you do need an SLO on those development environments. Otherwise your developers aren't going to be productive. And, you know, there is a, there is a cost to your organization as a result. And so embedding those practices early is, is actually a really, a really good way to build

new products. You know, I talk from a platform engineering perspective, but if you're building an application again, you can start to measure some of these things and you automate some of these processes in pre, in your pre-production environments, rather than leaving it until the sort of last thing you do before, before you go live. These are things that you can really bake into your engineering process. And I will assume that a fair amount of like, you know, folks that you're working with are using, working with the Tanzu portfolio, but it sounds like that, like there's any specific technology that's, you know, that anyone needs to do, obviously, because everyone's using different things, but it's just, it's, it's more of the practices around automation. Well, yeah, exactly. Yeah. So, I mean, our toolkit generally involves Terraform for provisioning. We use Concourse. We, we've used Prometheus and Grafana for observability. We're looking at Tanzu observability and Wavefront now that we've joined forces as part of VMware Tanzu. So, but, but yeah, there are, there are, there are, there are lots of options. There are lots of options. We have our go-to favorites and every engineering team has the same. So, so out of curiosity, what is your favorite part of SLE or what is it that really draws you? Why, why, why is it that you're so passionate about this? And maybe, maybe we already answered it, but I just wanted to get that specifically, because I do think it's important, like, you know, when people like, what is that thing that really, you know, it gets the why, like the, the personal touch, I guess, of why this thing matters. Yeah. I got involved in the DevOps community to, to start with, because I saw that good engineering practices made the environment for the humans working in software development so much better. You know, it was about the health and wellbeing of my own team to start with. Like, how can we get out of this cycle of, you know, rushing towards like three monthly release dates, having that enormous crunch of like testing and fixing at the end. And, you know, I started to research

continuous delivery and that's how, that's how I discovered DevOps. That's how I, I got interested in automation tooling and cloud, all of these things come together to actually make the life for the average software engineer better. And that's, that's what really matters to me because I've seen the impact of, of bad practices on, on people. I've seen burnouts. I've had, you know, I've had engineers on call, you know, having relentless, sleepless nights because of fragile systems in production. And that hurts like that, that hurt me as a manager, but it hurt my team more. Like it hurt their families. It hurt their relationships. Like it's, it's a very human benefit to getting these things right. And that's, that's why I, that's, that's, that's why my career took this direction. That's why I'm here doing this stuff and teaching our customers today, because I really do think that the teams who adopt these practices are, are going to be happy and healthier and more sustainable. And I, I've heard it. I've, I've heard, and I've seen DevOps compared to like SRE and DevOps compared, but it's not really comparison. I mean, it's, it seems like more of an evolution of, or, or maybe I know James Urquhart on our team has talked about like making it more like, like, like SRE is just part of a new generation of DevOps, maybe as a way of thinking about it, just as we rethink these things as we get on. So, you know, it's not DevOps or SRE doesn't seem like. Oh no, no, they're not opposed to the complementary. So DevOps is, I, I like to say it's more of a, more of a movement than it is a, like a tangible set of tools and practices. And SRE is a bit more of an opinionated implementation of DevOps. It's really, it's really about how to break down those silos and those, you know, those traditionally very, very conflicting priorities of a dev team who want to ship new features quickly and an ops team who need to protect reliability and prioritize

stability. Yeah. All right. And then I want to ask, I mean, getting to that too, I mean, going back to the, your statement about just kind of how human some of the benefits of this are and, and like the, how it improves the personal lives of, of the people tasked with, with keeping apps or systems up, like, what does that look, I mean, have you seen like the night and day sort of transition, or if we had to give like a very specific analogy to someone, I mean, I'm just curious how stark the difference can be if you've implemented this properly and then what that looks like in real life. Yeah. I mean, I, I have a, I have a really good example from, from the days of Pivotal before we joined VMware. So Pivotal cloud ops were really fighting fires constantly with, with the Pivotal web services platform. They were really suffering from alert fatigue. They had high attrition. They couldn't keep people in the team. They didn't have any time to make improvements and get anything done. My, my first conversation with this cloud ops team was actually alongside pager duty and pager duty was sort of presenting a different view of their data back to them. It was called the operations health dashboard. And this view of the data showed them how often they were getting interrupted during the night, how often they were getting interrupted during the evenings. You know, it was really trying to create a health score as like an aggregated score based on, based on these type of metrics about how a person's life was probably impacted by that alert, you know, things like, was it a false alarm as well? And, and, and yeah, like talking to that team and seeing the progress they make, you know, you start, you start looking at like very, very low health scores in that team, you know, that they will tell you themselves that they were struggling. Like they had a health score of less than 20. And over the course

of 12 months, they gradually implemented some of these SRE practice that I'm talking about. And, you know, members of that team are huge advocates for these practices today. Corey Innes has joined the CRE team here at VMware Tanzu and is, and is advocating for these practices with our customers as well. But, you know, these we've seen, we've seen it with, with a team that we're close to, that we know, and who we've worked with, how these, how these things have tangibly improved their wellbeing and improve the health of that team. All right. So, I mean, in closing, if you had to, because I think this is all super interesting. I think hearing about these concepts, if you're not familiar with them, or if you've heard about them and like something like SRE, but just haven't really been able to, you know, consume and take in what it means. I mean, that's obviously a huge part of it, but, but also just like, what would be the first, what's, I mean, what's a first step, right? If I'm someone and I'm like, okay, my team needs to adopt these things. These are going off the rails or, or whatever, or, you know, we're spending too much time trying to keep this thing up all the time. Like, like, like what's a first step? What is a tangible thing that an organization or a team lead can do to, to get moving in this direction? So, so, I mean, I'm going to give you two things because I'm awkward. So I think like, look at your team. What are, what are they struggling with? Are they struggling with alert fatigue? And is it that production support that they, that they are suffering with? In which case I would really like, my first step would be to look for a, a fee, a small handful of service level indicators, something that you could measure that indicates whether or not your users are happy. And everything goes from there. Once you know, those SLIs, you can set SLOs, you can start measuring your error budget, but that, that first step is agreeing on what the important metrics to measure are. And so if, so if a team is suffering with, with the burnout from production support, I would say go in that direction. The other way I would go is, you know, look at your team and say, actually, are they just really suffering with toil?

Are they having to do the same manual tasks over and over and over again? You know, do we need to, do we need to be thinking about automating some of that away? And again, the very first step, measure it, make sure you're tracking it, you know, identify where, where the, where the toil is being creative and where you can have the highest impact with your automation. So those are two ways to get started. I've slightly cheated on the question. Sorry about that. Oh, that's fine. That's fine. Maybe it was an ignorant question or a naive question. There, there is that one thing. And then, and then finally, finally, finally, is where do you, because that, that your answer just piqued this question is where, who do you see typically leading the charge? Is this an ops thing? Is this an app team? Like, like who's actually typically in charge of driving a transition to SRE or at least instigating it. I, I definitely, I definitely see it more within the sort of ops mindset. I advocate for teams that are autonomous and they build and run their own solutions. So a platform team would build and support their platform and an app team would build and support their apps in production. So, so I think it tends to be the people within whatever domain it is, who have a bit of more of an ops focus, who can advocate for these practices most effectively. So platform ops teams, get on the ball. And app ops teams. Yes. Whatever, however, however it's labeled. Yes. All right. All right. Well, hey, listen, Hannah, thanks a lot. Super insightful. Thank you. It's been fun.

Tanzu Talk · 🍕 Get more pizza.