1 00:00:00,000 --> 00:00:05,100 Welcome, listeners, to the Cloud and Culture Podcast. I am Derek Harris from VMware. 2 00:00:05,700 --> 00:00:09,900 And I am Danielle Burrow from VMware. If this is your first time listening, 3 00:00:10,120 --> 00:00:16,200 here's the quick background. This podcast focuses on VMware Pivotal Labs experts and clients who 4 00:00:16,200 --> 00:00:21,300 share their experiences, tips, best practices, and opinions on app modernization and digital 5 00:00:21,300 --> 00:00:25,360 transformation. For a more thorough introduction, listen to a previous episode. 6 00:00:26,260 --> 00:00:31,480 Yes. And this week, we are joined by Hannah Foxwell from the Pivotal Labs team. And Hannah 7 00:00:31,480 --> 00:00:37,500 explains all about SRE, which some people might know as Site Reliability Engineering. And 8 00:00:37,500 --> 00:00:44,120 Hannah and I discuss some of the principles behind SRE. We discuss how SRE helps with the 9 00:00:44,120 --> 00:00:49,400 retention of employees by making their lives easier and reducing the toil they have to put up with, 10 00:00:49,480 --> 00:00:55,100 and just generally making sure your applications are up and running and you're not wasting a lot 11 00:00:55,100 --> 00:01:02,280 of cycles over engineering for high reliability. So very insightful discussion. And you will learn 12 00:01:02,280 --> 00:01:07,840 everything you ever needed to know about SRE now. All right, Hannah. So thanks for being on. Can you 13 00:01:07,840 --> 00:01:11,560 start with just telling listeners a little bit about yourself and your role at VMware? 14 00:01:12,100 --> 00:01:18,180 Yeah, absolutely. So I'm Hannah. I am the Director for Platform Services here at VMware Tanzu. 15 00:01:18,180 --> 00:01:25,980 So essentially, my team is the field engineering team. We exist to support the engineers within our 16 00:01:25,980 --> 00:01:31,560 customers who are picking up Kubernetes and these technologies for the first time. Our aim really is 17 00:01:31,560 --> 00:01:38,460 to build wildly successful platform teams with them. All right. And so, I mean, that's a broad charter, 18 00:01:38,560 --> 00:01:42,900 I think. You know, one thing I think that we want to talk about specifically this week, as it relates to 19 00:01:42,900 --> 00:01:48,560 that, obviously, we can touch on the topics as they're connected. But it's SRE, systems, or I guess 20 00:01:48,560 --> 00:01:52,900 it's site reliability engineering. Yes, site reliability engineering. I was going to say system, but it is 21 00:01:52,900 --> 00:01:58,120 site. So can you just walk, give listeners a quick walkthrough, like, you know, what is SRE in a 22 00:01:58,120 --> 00:02:05,420 nutshell, for anyone who's unfamiliar? So I always sort of give the original definition of SRE, and then I 23 00:02:05,420 --> 00:02:12,120 can give you my kind of take on it. So Ben Treanor, who was the founder of Google's SRE team, said that 24 00:02:12,120 --> 00:02:19,200 SRE is what happens when a software engineer is tasked with what used to be called operations. And I think, 25 00:02:19,260 --> 00:02:26,880 you know, when O'Reilly published the SRE book in 2016, like, SRE sort of launched into our 26 00:02:26,880 --> 00:02:32,820 consciousness as a tech community. And we picked up many of these practices, but it's evolved a lot 27 00:02:32,820 --> 00:02:37,380 since then. And there's a huge community around this. It's not just happening at Google, it's happening 28 00:02:37,380 --> 00:02:44,840 at tech companies all over the world. And I think for me, SRE is about these tools and practices that 29 00:02:44,840 --> 00:02:51,000 can help teams reason about, prioritize, and deliver the reliability that their users need. 30 00:02:51,460 --> 00:02:57,060 It's not necessarily about massive scale, highly distributed systems, or always achieving five 31 00:02:57,060 --> 00:03:03,880 nines of availability. It's about delivering exactly the right amount of reliability for your users, 32 00:03:04,160 --> 00:03:08,500 and balancing the benefit of that reliability about the cost of delivering it. 33 00:03:08,680 --> 00:03:12,460 Yeah. So I think that you hit upon that. It's like, it's not just, it's a set of principles, 34 00:03:12,460 --> 00:03:16,420 basically, right? It's not, like you said, it's not kind of, it doesn't have to be widely distributed 35 00:03:16,420 --> 00:03:20,200 systems. You don't have to be running Kubernetes in order to take advantage of this. It's really 36 00:03:20,200 --> 00:03:23,940 kind of almost a mindset shift in a way. Is that, is that fair? 37 00:03:24,420 --> 00:03:29,320 Yeah, absolutely. Absolutely. So, you know, to deliver any kind of software product, you know, 38 00:03:29,320 --> 00:03:34,340 that reliability is fundamental. But I think what SRE gives us is it gives us a really powerful 39 00:03:34,340 --> 00:03:40,440 language to talk about failure. It says, yes, reliability is fundamental, but failure is to be 40 00:03:40,440 --> 00:03:47,840 expected. It's normal. And, you know, we not only need to invest in the prevention of it, 41 00:03:47,840 --> 00:03:51,980 but we need the processes and practices to be prepared for it when it happens. 42 00:03:51,980 --> 00:03:57,740 Right. And so let's walk through some of the concepts around SRE, because again, like, I think 43 00:03:57,740 --> 00:04:03,280 the people are familiar with SLA always becomes like, we always, it seems like whenever any discussion 44 00:04:03,280 --> 00:04:07,760 of SRE, you have to define like how S, how an SLA or a service level agreement ties into this, 45 00:04:07,840 --> 00:04:12,400 but it's not that, right? I mean, we have service level, I mean, SRE talks about service level 46 00:04:12,400 --> 00:04:17,240 indicators and objectives. You can just kind of walk through the general jargon or, I guess, 47 00:04:17,300 --> 00:04:18,240 the terminology. 48 00:04:19,040 --> 00:04:24,760 Yeah, absolutely. And I think this is like, for me, this is one of the absolutely sort of 49 00:04:24,760 --> 00:04:33,560 most valuable tools in the SRE toolkit. It's this ability to set your SLIs, your service level 50 00:04:33,560 --> 00:04:40,200 indicators. And, you know, these are to be the sort of unambiguous metrics by which you measure your 51 00:04:40,200 --> 00:04:45,780 reliability. This is something that you can automate the measurement of. It's not subjective 52 00:04:45,780 --> 00:04:52,500 and really it needs to be sort of observed from outside your system. You need to get the user's real 53 00:04:52,500 --> 00:04:59,480 world view of the reliability and the availability. And so you set your SLIs, that's the what, 54 00:04:59,480 --> 00:05:05,500 like, what are we going to measure? What can we use to get a good view of our system's performance? 55 00:05:06,320 --> 00:05:11,120 And then you set an SLO, which is the target level that you want to achieve on that metric. And 56 00:05:11,120 --> 00:05:17,700 usually it's expressed in a number of nines. For example, three nines would be 99.9% is the amount 57 00:05:17,700 --> 00:05:19,920 of time that you want to be achieving your SLI. 58 00:05:21,120 --> 00:05:25,980 The other piece of jargon that comes along with this, which I think is really important, 59 00:05:25,980 --> 00:05:34,360 is the inverse of the SLO. So if you say that your SLO is three nines, then you have a 0.1% 60 00:05:34,360 --> 00:05:42,360 acceptable failure, and that is the error budget. So you set your SLO to be your objective in terms 61 00:05:42,360 --> 00:05:48,800 of reliability, and you measure that unambiguously with an SLI, a service level indicator. But what you 62 00:05:48,800 --> 00:05:54,700 also do is you measure the amount of times that you're not achieving that. You measure the amount 63 00:05:54,700 --> 00:06:00,100 of times you're failing, and you keep that within a certain acceptable budget. And if you start to 64 00:06:00,100 --> 00:06:04,300 look like you're exceeding that budget, you can actually change your behaviors. You can make 65 00:06:04,300 --> 00:06:09,480 decisions based on that, and based on the user's experience of reliability about whether or not you 66 00:06:09,480 --> 00:06:12,500 want to introduce more risk into your system at that time. 67 00:06:12,800 --> 00:06:15,800 And the one thing that always struck me when I've talked to folks about this is the idea that if 68 00:06:15,800 --> 00:06:21,700 if you're overshooting, so if you're never even coming close to hitting your error budget, 69 00:06:21,700 --> 00:06:28,600 it, right? Like maybe you've built too much reliability into the system. Is that Yeah, it 70 00:06:28,600 --> 00:06:33,100 seems counterintuitive, but it is a matter of my understanding is it's a matter of, like, how you 71 00:06:33,100 --> 00:06:37,300 want to spend your time. And if you're spending too much energy to keep at five nines, and really, 72 00:06:38,100 --> 00:06:44,700 like, really, it's not exactly really, it's not needed. And I do I do think that, you know, this, 73 00:06:45,000 --> 00:06:51,300 you know, some of these behaviors, they're driven by some of the some of the more legacy practices that we 74 00:06:51,700 --> 00:06:57,600 had in operations, like every outage is a disaster, there's nothing such thing as an acceptable outage, 75 00:06:57,600 --> 00:07:03,100 you know, and when you get into that mindset that every service interruption and every outage is, 76 00:07:03,100 --> 00:07:11,600 is a disaster, then you start to, you start to really over engineer your solutions to, to try and achieve 77 00:07:11,600 --> 00:07:19,100 the impossible, which is 100%. And, and I think these, these are tendencies that exist within all engineering 78 00:07:19,100 --> 00:07:24,800 teams, you know, preparing for the absolute worst case scenario over engineering solutions. And if 79 00:07:24,800 --> 00:07:30,000 you're achieving a very high level of availability, that really your users don't need, it means that 80 00:07:30,000 --> 00:07:37,040 you have, you have probably invested too much in it, whether that be through engineering time, whether 81 00:07:37,040 --> 00:07:42,240 that be through redundant resources, maybe you've built a lot of resilience into your system that really 82 00:07:42,240 --> 00:07:48,600 wasn't needed. Maybe it was through automation that you then need to maintain over the long term, 83 00:07:49,000 --> 00:07:54,900 and, and creates toil for your team. There are lots of ways that this excessive amounts of 84 00:07:54,900 --> 00:08:00,760 reliability actually cost you, not just in terms of, you know, the fact that you actually could be 85 00:08:00,760 --> 00:08:06,200 potentially shipping features faster to your users and taking a few more risks in the application 86 00:08:06,200 --> 00:08:07,700 software development lifecycle. 87 00:08:07,700 --> 00:08:11,500 So yeah, I think that's definitely a matter of, of balance. And the other thing too, before we, 88 00:08:11,500 --> 00:08:15,000 because I want to get into how, how we've seen customers using it and how you've worked with 89 00:08:15,000 --> 00:08:18,660 organizations to, to adopt it. But I do think, right. The one thing I've also heard, I've talked, 90 00:08:18,740 --> 00:08:22,660 you know, speaking with folks from Google or whomever about it over the years is the other thing is, 91 00:08:22,660 --> 00:08:26,960 yes, like getting in terms of where you're spending your money thinking about, or your time and energy 92 00:08:26,960 --> 00:08:32,040 and toil is not just like over engineering, but also figuring out what you actually need to be tracking. 93 00:08:32,640 --> 00:08:37,160 So it used to be, we would track, you might be, you might've had 700 alerts on, right. 94 00:08:37,600 --> 00:08:38,340 Oh my God. 95 00:08:38,440 --> 00:08:43,020 But like six of them are the ones that actually matter. Right. I think that was the other big 96 00:08:43,020 --> 00:08:47,040 thing. And that just from the get-go that I think was a, for me, it always stands out when we talk 97 00:08:47,040 --> 00:08:52,360 with us, I read and get people familiar is like, it's about cutting down ideally or paring down what 98 00:08:52,360 --> 00:08:53,240 you're, what you're focusing on. 99 00:08:53,700 --> 00:08:58,020 Yeah, absolutely. And like the powerful thing about SLIs is it's really, it's really about, 100 00:08:58,020 --> 00:09:03,620 you know, what's important to my users. You know, we can monitor CPU and we can monitor error rates 101 00:09:03,620 --> 00:09:09,840 and things like that, but they're very difficult metrics to translate into real business impact or 102 00:09:09,840 --> 00:09:16,780 real user impact. And so if you, if you spend a bit more effort, you monitor less metrics, but you 103 00:09:16,780 --> 00:09:22,780 monitor the metrics that matter, you know, you don't end up with this scenario that we see so often, 104 00:09:22,780 --> 00:09:27,900 which is like alert fatigue. Yeah. You know, your page is going off every night. Yeah. You're not 105 00:09:27,900 --> 00:09:32,220 getting the sleep that you need to be. You're getting interrupted in your like evenings with your 106 00:09:32,220 --> 00:09:37,260 family. And for no reason, there was no service interruption and your users didn't care. 107 00:09:37,560 --> 00:09:43,140 I think of that. I used to have a ring doorbell and it would go off when like the wind blew. 108 00:09:43,140 --> 00:09:46,360 And then, so you'd get these, the chime would go off all the time and eventually just ignore it. 109 00:09:46,640 --> 00:09:50,920 So something must happen. Yeah. Oh my God. I might get set off by spiders. It's the most, 110 00:09:50,980 --> 00:09:58,420 it's the most terrifying thing. Yeah. That's, that's my, that's my everyday example of alert fatigue. 111 00:09:58,460 --> 00:10:01,940 No, no. It's the perfect example though, isn't it? It's the perfect example. 112 00:10:02,600 --> 00:10:05,480 It's like, yeah, I want to know when someone's actually at the door that, that I care about. 113 00:10:06,580 --> 00:10:12,980 So, so I wanted to ask for, in terms of like customers and, and, you know, pivotal labs clients, 114 00:10:12,980 --> 00:10:18,900 I'm curious to hear kind of how you've worked with them or maybe, I don't know, this is probably, 115 00:10:18,980 --> 00:10:23,200 this is probably too broad to start with, but so feel free to jump in where you want. But I mean, 116 00:10:23,200 --> 00:10:26,340 I'm curious about like, you know, where they, I'm curious about success stories, obviously, 117 00:10:26,340 --> 00:10:30,560 but also like where they've run into issues and, and kind of, you know, the real world where the 118 00:10:30,560 --> 00:10:34,160 rubber hits the road of doing this. Cause the terminology is one thing and the concepts are one thing, 119 00:10:34,160 --> 00:10:38,240 but then doing it obviously, you know, probably a whole other ball game. So. 120 00:10:38,240 --> 00:10:42,700 Yeah. I mean, I could, I could talk about the, the challenges to adopting this sort of thing 121 00:10:42,700 --> 00:10:48,000 all day. I'll talk about, I'll talk about a couple of things that I consider to be success stories 122 00:10:48,000 --> 00:10:55,560 though. So I think one thing about defining user centric SLIs that, that really changes the 123 00:10:55,560 --> 00:11:03,060 perception of, of systems is, is, is that actually in a lot of organizations, there are no 124 00:11:03,060 --> 00:11:09,660 unambiguous metrics for delivering availability. And when I was talking to, to some customers who had a 125 00:11:09,660 --> 00:11:16,060 very sort of traditional mindset, it turned out that their view of availability was really subjective. 126 00:11:16,060 --> 00:11:22,020 And it was really about, you know, the service manager was like negotiating the number of minutes 127 00:11:22,020 --> 00:11:27,400 downtime that they would record against a certain incident. And, you know, some incidents weren't even 128 00:11:27,400 --> 00:11:34,000 recorded. Like some of them were like, Oh, that was outside of your control. So that doesn't count, 129 00:11:34,000 --> 00:11:40,480 but it still impacted users. So this, this is, this is what troubled me about it. So you, they were 130 00:11:40,480 --> 00:11:48,880 achieving very, very, very high levels of availability on the face of it. But behind the scenes, there was 131 00:11:48,880 --> 00:11:56,240 this person to person negotiation of what counted as a service interruption. It wasn't about the user's 132 00:11:56,240 --> 00:12:02,260 perception. It was about the relationships and whose fault it was and which system went down. And I think, 133 00:12:02,480 --> 00:12:08,140 you know, that just, that just prevents you from having good conversations about where you need to 134 00:12:08,140 --> 00:12:15,000 invest in better resiliency and better practices. But, but yeah, so I introduced, I introduced this one team, 135 00:12:15,000 --> 00:12:22,480 like the place where this anecdote came from to the concept of like SLIs and SLOs. And they started 136 00:12:22,480 --> 00:12:28,820 measuring the availability of that one system that I was working with them on unambiguously for the 137 00:12:28,820 --> 00:12:34,200 first time. And of course, like it wasn't what they thought it would be. It didn't, it didn't compare 138 00:12:34,200 --> 00:12:39,880 at all to all these other systems where they were achieving very high levels of availability, but of course 139 00:12:39,880 --> 00:12:45,800 they weren't counting every outage as being real or valid. And so I think just introducing these 140 00:12:45,800 --> 00:12:52,640 different ways of, of talking about it in, in small pockets within an organization can help set a 141 00:12:52,640 --> 00:12:58,800 really good example of, you know, like here are our stats and we can absolutely stand behind them. 142 00:12:58,880 --> 00:13:04,120 We know that we've delivered this. And we also like, we know when we've overachieved and we know when 143 00:13:04,120 --> 00:13:10,360 we've underachieved and when we've underachieved, we've invested in like these three things to improve 144 00:13:10,360 --> 00:13:15,280 our resiliency and we got better results and we can show you that we got better results because the 145 00:13:15,280 --> 00:13:20,520 data is unambiguous. It's measured from a user point of view. Interesting. So, so it almost sounds 146 00:13:20,520 --> 00:13:26,120 like it's like, like it's might be a cliche, but it's a cultural thing to some degree. Oh, it is. 147 00:13:26,120 --> 00:13:32,120 Yeah. Okay. All right. Because the technological fix might be, like you said, changing what you're 148 00:13:32,120 --> 00:13:37,240 measuring and setting up, you know, setting up reliability in areas where it actually matter, you 149 00:13:37,240 --> 00:13:41,960 know, that matter, it affects the user experience. But in the end of the day, it's like, if the 150 00:13:41,960 --> 00:13:47,400 organization cannot come around to changing its mindset around placing blame or, you know, assigning 151 00:13:47,400 --> 00:13:52,200 blame, I should say, or around. Oh yeah. Well, I was going to say, like the challenges that I see are 152 00:13:52,200 --> 00:13:58,280 exactly those things. It's the, it's, it's some of the kind of fear, fear of production, 153 00:13:58,840 --> 00:14:05,160 fear of outages, fear of instability. And, and, you know, also the, also that sort of blame culture 154 00:14:05,160 --> 00:14:13,560 where, you know, you will, you will sort of identify a root cause or a team that did wrong or a person 155 00:14:13,560 --> 00:14:18,280 that did wrong. And, you know, those, those incidences, if they're managed like that can be really 156 00:14:18,280 --> 00:14:22,840 traumatic, they can make you feel like you don't want to talk about something that went wrong. 157 00:14:22,840 --> 00:14:26,760 And of course, if you don't talk about it, if you hide it, you can't learn from it and improve it. 158 00:14:28,120 --> 00:14:31,880 Yeah. Actually, I mean, yeah, the, the blame thing is I just heard T-Mobile talking about this 159 00:14:31,880 --> 00:14:35,720 at SpringOne a couple of months ago, and they're talking about how they handle communication when 160 00:14:35,720 --> 00:14:40,920 there's an outage. And there was the big thing was not trying not to assign blame, like fixing the 161 00:14:40,920 --> 00:14:45,720 problem. And then, you know, if there needs to be some sort of responsibility, whatever, 162 00:14:45,720 --> 00:14:49,160 like you deal with that later. But the first thing was just fixing the problem. But I think, 163 00:14:49,160 --> 00:14:53,560 yeah, that gets to, yeah, just, just generally speaking, this idea of always needing to assign 164 00:14:53,560 --> 00:14:58,360 blame. And it's just a cultural mind shift that, that gets inhibits a lot of, I think cloud native, 165 00:14:58,360 --> 00:15:03,000 if I consider SRE to be a cloud native practice in a matter of speaking, I think it. 166 00:15:03,000 --> 00:15:08,120 Yeah. I've actually like talking, talking about blame. I've had, I've had somewhat of the opposite 167 00:15:08,120 --> 00:15:12,760 scenario as well. So I was talking to one customer about how they might, they might adopt some of these 168 00:15:12,760 --> 00:15:19,400 practices in their team. And they, they had, they had quite a legacy, fragile system that they were 169 00:15:19,400 --> 00:15:25,320 dependent on. A lot of their services were dependent on like the availability and reliability of this 170 00:15:25,320 --> 00:15:32,440 legacy system. And they felt almost like almost powerless in this situation that they couldn't 171 00:15:32,440 --> 00:15:38,200 improve their own performance and their own availability, because it was always, it was always 172 00:15:38,200 --> 00:15:43,560 degraded by this other system. And I said, well, the first thing you can do is so you could still 173 00:15:43,560 --> 00:15:48,760 start measuring it because then you have a business case to invest in this other system. You know, 174 00:15:48,760 --> 00:15:53,720 reliability is a team sport in this case. Like you don't, you're not delivering it in isolation. 175 00:15:54,520 --> 00:15:59,400 So yeah, I mean, a conversation that maybe started with a bit of blame, like, oh, we can't, 176 00:15:59,400 --> 00:16:03,480 we're helpless. We can't do anything about it. It's their fault. Actually, you know, I think it's, 177 00:16:03,480 --> 00:16:13,080 it's still a good idea to measure the actual availability of your system because it's not 178 00:16:13,080 --> 00:16:17,720 possible sometimes to deliver that reliability that your users need in isolation. And as long 179 00:16:17,720 --> 00:16:23,160 as you keep your users' needs in mind, you should be able to make improvements and make sure the 180 00:16:23,160 --> 00:16:24,600 investment goes to the right places. 181 00:16:24,600 --> 00:16:29,320 Yeah. I like that. Reliability is a team sport. That's like, that's a T-shirt. 182 00:16:30,760 --> 00:16:31,080 Yes. 183 00:16:31,080 --> 00:16:37,000 Do you have, do you have a, I mean, the other thing I'm always curious about is whether 184 00:16:37,640 --> 00:16:43,640 organizations can, SRE is something you can piecemeal and pick and choose the pieces that 185 00:16:43,640 --> 00:16:47,480 make sense. Cause I've heard, I've heard some folks talk about it, but then they just, you know, 186 00:16:47,480 --> 00:16:51,000 they said, but we're not doing this, this, this. And I found myself going, are you really doing it 187 00:16:51,000 --> 00:16:54,360 then? So, so can you pick and choose what works or do you kind of have to buy in? 188 00:16:54,600 --> 00:16:57,240 Whole hog and, and really embrace it. 189 00:16:57,240 --> 00:17:03,400 I mean, you're not going to, you're not going to learn all of this new stuff and implement 190 00:17:03,400 --> 00:17:09,400 everything overnight. So I think as long as you have an intent to start and continuously improve, 191 00:17:09,400 --> 00:17:16,760 then you're doing all right. You know, I can talk about teams who don't use SLIs and SLOs, 192 00:17:16,760 --> 00:17:22,040 but they do do blameless postmortems on, on their incidents. They do create that blameless space 193 00:17:22,040 --> 00:17:27,160 where not every outage is a disaster and it's an opportunity to learn and improve. So that, 194 00:17:27,160 --> 00:17:34,200 that in isolation delivers a, a, an amount of value. And also, you know, when we talk about 195 00:17:34,200 --> 00:17:43,000 eliminating toil and using automation to do that, to, to, to build a software solution to what would be 196 00:17:43,000 --> 00:17:50,280 like a manual human repetitive task, you know, that's again, something that has value as a standalone 197 00:17:50,280 --> 00:17:56,600 practice. You know, you can reproduce more consistent environments using infrastructure as 198 00:17:56,600 --> 00:18:03,320 code and configuration management systems. You know, you can, you can rebuild your pre-production 199 00:18:03,320 --> 00:18:10,120 environments overnight. If you script it in the right way, these things improve the consistency and 200 00:18:10,120 --> 00:18:15,240 reliability of, of, of those things in isolation, but you're not necessarily going to get all of 201 00:18:15,240 --> 00:18:18,440 the benefits that all of the other SRE practices bring you. 202 00:18:18,440 --> 00:18:23,640 So, so, so what are the like reliability, increased reliability aside, which obviously 203 00:18:24,360 --> 00:18:28,360 is a big deal. I mean, cause you mentioned toil a few times. So I'm, I'm, and, and I think reducing 204 00:18:28,360 --> 00:18:33,000 toil is, is, is one of the big, the, the big benefits. Okay. So can you talk about, I mean, 205 00:18:33,000 --> 00:18:38,440 let's say reliability aside, what are the, what are the benefits that, that clients are seeing as 206 00:18:38,440 --> 00:18:43,560 as they're, as they're embracing SRE? Wow. Yeah. Let's, let, let's talk about toil, 207 00:18:43,560 --> 00:18:49,080 because this is, this is one of the, one of the reasons that I took an interest in SRE to start 208 00:18:49,080 --> 00:18:58,440 with, because I was, I was, I was working on a platform team of a large e-commerce retailer in the UK. 209 00:18:59,000 --> 00:19:04,840 And we got ourselves into a situation where, you know, there was so much toil that there was no 210 00:19:04,840 --> 00:19:11,480 longer time in the day to automate the things that we knew we needed to. There was so much manual work 211 00:19:11,480 --> 00:19:16,360 to keep the lights on, as you might say, that, that even though we knew the best practices, and even 212 00:19:16,360 --> 00:19:22,840 though we wanted to do the right thing, we didn't have enough time. And so I think, I think the, 213 00:19:24,040 --> 00:19:30,200 the, the sort of Google kind of standard around this is to keep, keep an eye on your tool, toil and cap 214 00:19:30,200 --> 00:19:37,240 it at 50%. Like if you cap it and it starts to get towards that 50%, like we're spending half our time 215 00:19:37,240 --> 00:19:43,560 on sort of manual work or work that has no enduring value, then we're going to get ourselves into a 216 00:19:43,560 --> 00:19:50,920 situation where we no longer have time to do it. And all we do is add more humans to do more of this 217 00:19:50,920 --> 00:19:58,600 manual work. As a, as a fantastic quote in, in the SRE book, which is, which says that if we have to 218 00:19:58,600 --> 00:20:03,480 starve humans to do the work, we're feeding the machines with the blood, sweat and tears of human 219 00:20:03,480 --> 00:20:09,400 beings, which I think is a fantastic quote. And it sort of some, sums up my experience of, 220 00:20:09,960 --> 00:20:15,160 of what not to do when I was on the other side and when I was managing a platform team myself. 221 00:20:16,360 --> 00:20:21,640 Yeah. And it just brings to, brings to mind every, like disgruntled sysadmin. 222 00:20:24,200 --> 00:20:30,680 Absolutely. Yeah. Like, and you know, it's, it's such a stereotype within so many organizations that, 223 00:20:30,680 --> 00:20:35,640 you know, the, like the ops team are just overwhelmed with keeping the lights on that 224 00:20:35,640 --> 00:20:40,600 they don't have any time for continuous improvement, but it's so true. And that, and measuring and managing 225 00:20:40,600 --> 00:20:46,280 your toil is, is one way to keep, keep that in check and make sure you don't go down that slippery slope. 226 00:20:47,080 --> 00:20:52,760 Yeah. Well, and I assume, and I'd love to hear if you've seen this in action as well, but like as, 227 00:20:53,800 --> 00:20:57,880 as the world becomes more digital and as every, you know, software becomes more important within 228 00:20:57,880 --> 00:21:02,680 companies and, and more traditional companies are competing maybe with tech companies for talent, 229 00:21:02,680 --> 00:21:08,200 even like, like if, if, if you're going into a place where that's going to be your day to day, 230 00:21:08,200 --> 00:21:11,000 you know, and you're a talented person, you might not stay long. 231 00:21:11,000 --> 00:21:16,120 No. So I think there's a degree like, yeah, like, so like life, your work life balance or your work 232 00:21:16,120 --> 00:21:23,480 experience is actually going to matter for the company in terms of retention or in terms of attracting 233 00:21:23,480 --> 00:21:27,960 talent. And that's, that's not a small deal when you're, when you're competing with companies like 234 00:21:27,960 --> 00:21:29,080 Google or Facebook who will. 235 00:21:29,080 --> 00:21:33,720 No, absolutely not. And, you know, like, you know, every, every sort of, 236 00:21:33,720 --> 00:21:38,360 every software engineering team has something to offer, but you have to think, you know, there's, 237 00:21:38,360 --> 00:21:44,200 there's a lot of companies who are paying, paying very highly in this space at the moment, 238 00:21:44,200 --> 00:21:49,160 especially now that it's a global workforce and we're all working in a very distributed way. 239 00:21:49,160 --> 00:21:54,840 You know, what are you going to do to make sure you keep the talent, you know, and if you can't 240 00:21:54,840 --> 00:22:00,440 compete on maybe salaries, then you have to, you have to offer something else. You have to offer that, 241 00:22:01,000 --> 00:22:06,520 you know, benefits in terms of health and, you know, using, using best practices and giving the 242 00:22:06,520 --> 00:22:12,200 engineers the tools they need to do, to do their job and to do more interesting problem solving and 243 00:22:12,200 --> 00:22:17,400 personal development than if they're just holding the pager, keeping the lights on. 244 00:22:18,200 --> 00:22:21,160 Right. And I say compete with Google and Facebook, but realistic, I mean, the more, 245 00:22:21,160 --> 00:22:25,720 the more realistic thing is probably competing with other large companies in your own city or in your 246 00:22:26,360 --> 00:22:29,560 geography or whatever. It's not, it's not just the big companies anymore. And that's kind of the 247 00:22:30,360 --> 00:22:30,680 important thing. 248 00:22:30,680 --> 00:22:37,080 Yeah. Absolutely. Yes. Everyone. You're right. Yeah. Yeah. Are there, are there technological 249 00:22:37,080 --> 00:22:43,160 solutions that come into play? I mean, I know like, again, SRE is not, you know, part and parcel 250 00:22:43,880 --> 00:22:47,960 a must have for digital transformation. You don't need to be using Kubernetes. You don't need to, 251 00:22:47,960 --> 00:22:53,080 like, like these, all these buzzwords do not necessarily, or buzzwords is kind of pejorative, 252 00:22:53,080 --> 00:22:58,920 but all these, all these things do not necessarily overlap with one another, but are there, 253 00:22:58,920 --> 00:23:04,680 do you see certain technologies or certain practices that, or, or, you know, or processes or tools or 254 00:23:04,680 --> 00:23:09,320 whatever that actually do tend to help out a lot when, when trying to adopt us or re-principles? 255 00:23:11,160 --> 00:23:18,520 Yeah. I think, you know, I've, the type of work that I do with my team is very much about building, 256 00:23:18,520 --> 00:23:27,560 building platforms. And, you know, we, we tend to, we, we tend to skip ahead somewhat because of our 257 00:23:27,560 --> 00:23:34,280 years of experience to, to automation, because we know that if you don't automate from, from day one, 258 00:23:34,280 --> 00:23:38,360 you're going to have some debts that you need to pay off in the future to keep on top of your 259 00:23:38,360 --> 00:23:44,120 toil. So we almost shortcut, shortcut that process of introducing automation on day one. 260 00:23:44,120 --> 00:23:50,200 But what we also like to do is we also like to measure our success in, in building these platforms 261 00:23:50,200 --> 00:23:56,520 unambiguously. And so that requires a tool, some tools in place to, to measure your service level 262 00:23:56,520 --> 00:24:01,240 indicators. And, you know, a lot of people think that these things are only valid in production, 263 00:24:01,240 --> 00:24:05,400 but they're valid in pre-production as well. You're not going to be aiming for five nines 264 00:24:05,400 --> 00:24:11,640 availability in your pre-production environment, but there is a cost associated with them going down. 265 00:24:11,640 --> 00:24:18,200 And so, you know, you do need an SLO on those development environments. Otherwise your developers 266 00:24:18,200 --> 00:24:25,000 aren't going to be productive. And, you know, there is a, there is a cost to your organization as a 267 00:24:25,000 --> 00:24:34,040 result. And so embedding those practices early is, is actually a really, a really good way to build 268 00:24:34,040 --> 00:24:39,080 new products. You know, I talk from a platform engineering perspective, but if you're building an 269 00:24:39,080 --> 00:24:43,880 application again, you can start to measure some of these things and you automate some of these 270 00:24:43,880 --> 00:24:47,720 processes in pre, in your pre-production environments, rather than leaving it until 271 00:24:47,720 --> 00:24:52,840 the sort of last thing you do before, before you go live. These are things that you can really bake 272 00:24:52,840 --> 00:24:59,240 into your engineering process. And I will assume that a fair amount of like, you know, folks that you're 273 00:24:59,240 --> 00:25:05,400 working with are using, working with the Tanzu portfolio, but it sounds like that, like there's 274 00:25:05,400 --> 00:25:09,720 any specific technology that's, you know, that anyone needs to do, obviously, because everyone's using 275 00:25:09,720 --> 00:25:12,760 different things, but it's just, it's, it's more of the practices around automation. 276 00:25:12,760 --> 00:25:18,440 Well, yeah, exactly. Yeah. So, I mean, our toolkit generally involves Terraform for provisioning. 277 00:25:18,440 --> 00:25:25,240 We use Concourse. We, we've used Prometheus and Grafana for observability. We're looking at 278 00:25:25,240 --> 00:25:30,040 Tanzu observability and Wavefront now that we've joined forces as part of VMware Tanzu. 279 00:25:30,040 --> 00:25:35,400 So, but, but yeah, there are, there are, there are, there are lots of options. There are lots of 280 00:25:35,400 --> 00:25:39,080 options. We have our go-to favorites and every engineering team has the same. 281 00:25:39,080 --> 00:25:45,320 So, so out of curiosity, what is your favorite part of SLE or what is it that really 282 00:25:46,360 --> 00:25:49,800 draws you? Why, why, why is it that you're so passionate about this? And maybe, maybe we already 283 00:25:49,800 --> 00:25:53,240 answered it, but I just wanted to get that specifically, because I do think it's important, 284 00:25:53,240 --> 00:25:57,880 like, you know, when people like, what is that thing that really, you know, it gets the why, 285 00:25:57,880 --> 00:26:01,240 like the, the personal touch, I guess, of why this thing matters. 286 00:26:02,200 --> 00:26:11,560 Yeah. I got involved in the DevOps community to, to start with, because I saw that good engineering 287 00:26:11,560 --> 00:26:20,440 practices made the environment for the humans working in software development so much better. 288 00:26:20,440 --> 00:26:25,320 You know, it was about the health and wellbeing of my own team to start with. Like, how can 289 00:26:25,320 --> 00:26:32,680 we get out of this cycle of, you know, rushing towards like three monthly release dates, having 290 00:26:32,680 --> 00:26:37,560 that enormous crunch of like testing and fixing at the end. And, you know, I started to research 291 00:26:37,560 --> 00:26:43,240 continuous delivery and that's how, that's how I discovered DevOps. That's how I, I got interested 292 00:26:43,240 --> 00:26:51,320 in automation tooling and cloud, all of these things come together to actually make the life for the 293 00:26:51,320 --> 00:26:56,600 average software engineer better. And that's, that's what really matters to me because I've seen the 294 00:26:56,600 --> 00:27:03,480 impact of, of bad practices on, on people. I've seen burnouts. I've had, you know, I've had engineers 295 00:27:03,480 --> 00:27:09,320 on call, you know, having relentless, sleepless nights because of fragile systems in production. 296 00:27:09,320 --> 00:27:15,240 And that hurts like that, that hurt me as a manager, but it hurt my team more. Like it hurt 297 00:27:15,240 --> 00:27:23,400 their families. It hurt their relationships. Like it's, it's a very human benefit to getting these 298 00:27:23,400 --> 00:27:28,040 things right. And that's, that's why I, that's, that's, that's why my career took this direction. 299 00:27:28,040 --> 00:27:33,720 That's why I'm here doing this stuff and teaching our customers today, because I really do think that 300 00:27:33,720 --> 00:27:39,800 the teams who adopt these practices are, are going to be happy and healthier and more sustainable. 301 00:27:40,840 --> 00:27:46,040 And I, I've heard it. I've, I've heard, and I've seen DevOps compared to like SRE and DevOps compared, 302 00:27:46,040 --> 00:27:50,520 but it's not really comparison. I mean, it's, it seems like more of an evolution of, or, or maybe 303 00:27:50,520 --> 00:27:56,760 I know James Urquhart on our team has talked about like making it more like, like, like SRE is just 304 00:27:56,760 --> 00:28:01,240 part of a new generation of DevOps, maybe as a way of thinking about it, just as we rethink these 305 00:28:01,240 --> 00:28:05,240 things as we get on. So, you know, it's not DevOps or SRE doesn't seem like. 306 00:28:05,240 --> 00:28:12,040 Oh no, no, they're not opposed to the complementary. So DevOps is, I, I like to say it's more of a, 307 00:28:12,040 --> 00:28:20,040 more of a movement than it is a, like a tangible set of tools and practices. And SRE is a bit more 308 00:28:20,040 --> 00:28:26,920 of an opinionated implementation of DevOps. It's really, it's really about how to break down those 309 00:28:26,920 --> 00:28:33,640 silos and those, you know, those traditionally very, very conflicting priorities of a dev team 310 00:28:33,640 --> 00:28:40,440 who want to ship new features quickly and an ops team who need to protect reliability and prioritize 311 00:28:40,440 --> 00:28:46,840 stability. Yeah. All right. And then I want to ask, I mean, getting to that too, I mean, 312 00:28:46,840 --> 00:28:53,080 going back to the, your statement about just kind of how human some of the benefits of this are and, 313 00:28:53,080 --> 00:28:58,760 and like the, how it improves the personal lives of, of the people tasked with, with keeping apps or 314 00:28:58,760 --> 00:29:03,960 systems up, like, what does that look, I mean, have you seen like the night and day sort of transition, 315 00:29:03,960 --> 00:29:09,160 or if we had to give like a very specific analogy to someone, I mean, I'm just curious how stark the 316 00:29:09,160 --> 00:29:14,760 difference can be if you've implemented this properly and then what that looks like in real life. 317 00:29:15,720 --> 00:29:21,640 Yeah. I mean, I, I have a, I have a really good example from, from the days of Pivotal before we 318 00:29:21,640 --> 00:29:30,280 joined VMware. So Pivotal cloud ops were really fighting fires constantly with, with the Pivotal web 319 00:29:30,280 --> 00:29:36,600 services platform. They were really suffering from alert fatigue. They had high attrition. They couldn't keep 320 00:29:36,600 --> 00:29:42,200 people in the team. They didn't have any time to make improvements and get anything done. 321 00:29:43,160 --> 00:29:50,520 My, my first conversation with this cloud ops team was actually alongside pager duty and pager duty 322 00:29:50,520 --> 00:29:55,880 was sort of presenting a different view of their data back to them. It was called the operations 323 00:29:55,880 --> 00:30:03,320 health dashboard. And this view of the data showed them how often they were getting interrupted during 324 00:30:03,320 --> 00:30:08,520 the night, how often they were getting interrupted during the evenings. You know, it was really trying to 325 00:30:08,520 --> 00:30:15,960 create a health score as like an aggregated score based on, based on these type of metrics about how 326 00:30:15,960 --> 00:30:22,920 a person's life was probably impacted by that alert, you know, things like, was it a false alarm as well? 327 00:30:22,920 --> 00:30:30,360 And, and, and yeah, like talking to that team and seeing the progress they make, you know, you start, 328 00:30:31,080 --> 00:30:36,440 you start looking at like very, very low health scores in that team, you know, that they will tell 329 00:30:36,440 --> 00:30:41,400 you themselves that they were struggling. Like they had a health score of less than 20. And over the course 330 00:30:41,400 --> 00:30:47,800 of 12 months, they gradually implemented some of these SRE practice that I'm talking about. And, you know, 331 00:30:47,800 --> 00:30:55,240 members of that team are huge advocates for these practices today. Corey Innes has joined the CRE team 332 00:30:55,240 --> 00:31:00,520 here at VMware Tanzu and is, and is advocating for these practices with our customers as well. 333 00:31:00,520 --> 00:31:06,360 But, you know, these we've seen, we've seen it with, with a team that we're close to, that we know, 334 00:31:06,360 --> 00:31:13,640 and who we've worked with, how these, how these things have tangibly improved their wellbeing and 335 00:31:13,640 --> 00:31:18,440 improve the health of that team. All right. So, I mean, in closing, if you had to, 336 00:31:19,480 --> 00:31:23,400 because I think this is all super interesting. I think hearing about these concepts, if you're 337 00:31:23,400 --> 00:31:27,000 not familiar with them, or if you've heard about them and like something like SRE, but just haven't 338 00:31:27,000 --> 00:31:31,480 really been able to, you know, consume and take in what it means. I mean, that's obviously 339 00:31:32,680 --> 00:31:37,320 a huge part of it, but, but also just like, what would be the first, what's, I mean, what's a first 340 00:31:37,320 --> 00:31:41,480 step, right? If I'm someone and I'm like, okay, my team needs to adopt these things. These are going 341 00:31:41,480 --> 00:31:45,640 off the rails or, or whatever, or, you know, we're spending too much time trying to keep this thing 342 00:31:45,640 --> 00:31:50,440 up all the time. Like, like, like what's a first step? What is a tangible thing that an organization 343 00:31:50,440 --> 00:31:56,760 or a team lead can do to, to get moving in this direction? So, so, I mean, I'm going to give you 344 00:31:56,760 --> 00:32:01,880 two things because I'm awkward. So I think like, look at your team. What are, what are they 345 00:32:01,880 --> 00:32:05,960 struggling with? Are they struggling with alert fatigue? And is it that production support that 346 00:32:05,960 --> 00:32:12,520 they, that they are suffering with? In which case I would really like, my first step would be to look 347 00:32:12,520 --> 00:32:19,000 for a, a fee, a small handful of service level indicators, something that you could measure that 348 00:32:19,000 --> 00:32:23,160 indicates whether or not your users are happy. And everything goes from there. Once you know, 349 00:32:23,160 --> 00:32:28,360 those SLIs, you can set SLOs, you can start measuring your error budget, but that, that first 350 00:32:28,360 --> 00:32:34,520 step is agreeing on what the important metrics to measure are. And so if, so if a team is suffering 351 00:32:34,520 --> 00:32:40,600 with, with the burnout from production support, I would say go in that direction. The other way I 352 00:32:40,600 --> 00:32:44,840 would go is, you know, look at your team and say, actually, are they just really suffering with toil? 353 00:32:44,840 --> 00:32:50,280 Are they having to do the same manual tasks over and over and over again? You know, do we need to, 354 00:32:50,280 --> 00:32:55,480 do we need to be thinking about automating some of that away? And again, the very first step, 355 00:32:55,480 --> 00:33:01,480 measure it, make sure you're tracking it, you know, identify where, where the, where the toil is 356 00:33:01,480 --> 00:33:08,040 being creative and where you can have the highest impact with your automation. So those are two ways 357 00:33:08,040 --> 00:33:11,320 to get started. I've slightly cheated on the question. Sorry about that. 358 00:33:11,320 --> 00:33:16,360 Oh, that's fine. That's fine. Maybe it was an ignorant question or a naive question. There, 359 00:33:16,360 --> 00:33:22,360 there is that one thing. And then, and then finally, finally, finally, is where do you, because that, 360 00:33:22,360 --> 00:33:28,200 that your answer just piqued this question is where, who do you see typically leading the charge? Is 361 00:33:28,200 --> 00:33:33,640 this an ops thing? Is this an app team? Like, like who's actually typically in charge of driving 362 00:33:33,640 --> 00:33:41,240 a transition to SRE or at least instigating it. I, I definitely, I definitely see it more within the 363 00:33:41,240 --> 00:33:49,720 sort of ops mindset. I advocate for teams that are autonomous and they build and run their own 364 00:33:49,720 --> 00:33:55,720 solutions. So a platform team would build and support their platform and an app team would build and 365 00:33:55,720 --> 00:34:01,960 support their apps in production. So, so I think it tends to be the people within whatever domain it is, 366 00:34:01,960 --> 00:34:08,600 who have a bit of more of an ops focus, who can advocate for these practices most effectively. 367 00:34:09,880 --> 00:34:15,560 So platform ops teams, get on the ball. And app ops teams. 368 00:34:16,440 --> 00:34:18,360 Yes. Whatever, however, however it's labeled. 369 00:34:20,120 --> 00:34:24,760 Yes. All right. All right. Well, hey, listen, Hannah, thanks a lot. Super insightful. 370 00:34:24,760 --> 00:34:29,240 Thank you. It's been fun.