How State Farm Teaches Developers New Tricks

Episode 200 · Tue, 20 Apr 2021 15:00:00 +0200 · 32:41 · SRT · archive.org · all Tanzu Talk

0:00

Welcome to another episode of Cloud and Culture. I'm Danielle Burrow from VMware. And I'm Derek Harris from VMware. If you're new, this podcast focuses on the process of application and IT modernization through the lens of Tanzu Labs, previously called Pivotal Labs. Tanzu Labs experts have helped hundreds of companies and organizations, big and small, build better software and adopt modern development practices. Our guest for this episode is Jeff Anderson, an architecture manager at State Farm. Jeff explains how he helped modernize the insurance leader's application platforms and how he now helps other teams get up to speed with new technologies and cloud-native practices like DevOps, SRE, and chaos engineering. He also discusses how the company decides which applications to run on which platforms, both on cloud and on-prem. So stick around to hear Jeff's insights and hopefully learn some lessons to accelerate your modernization efforts. So Jeff, thank you so much for joining. Would you mind just giving a quick introduction and tell us a little bit about your role at State Farm? Yeah, absolutely. And thank you for having me. This is exciting. Yeah, I'm Jeff Anderson. I've been at State Farm for over 20 years. I've had various roles throughout the years. You know, from the first thing I did, you know, managing applications on, you know, green screen, HP 3000, COBOL system for our claim system to installing and administering our first Java web app hosting infrastructure. And now my role is as an architecture manager helping roughly our 1800 or so developers understand cloud native and applying those principles and getting to the latest on and off-prem cloud infrastructure. Great. And could you, you know, maybe we can get started just by telling us a little bit about some of the, like, big picture transformation initiatives that are happening at State Farm and kind of what are some of the different teams or groups that are supporting that? Yeah. So for just to set a little bit of context, you know, I'll tell you a bit about like what we

2:04

have, technologies we've got, and then where we're going and the teams that are kind of helping get there. But, you know, obviously State Farm is not a startup, right? We're a hundred year old company. And like a lot of big companies, we have a wide spectrum of technologies. And I'll give you an idea of that, the landscape of that tech, right? So we have data centers with, with on-prem tech, you know, it's chock full of everything from mainframes to commodity x86, networks, network appliances, and a lot more. We have mainframes with tons of data on them. We have, you know, on, on the commodity, we have VMs with our own, like custom created legacy PaaS system. And, but we also have Tanzu application service and, and Kubernetes clusters available to, for our developers. And then also we have access to like public cloud providers as well. And, you know, the, the latter ones are the more strategic platforms for our developers. So it really, in order to modernize and get there, you know, the, the challenges that, you know, a huge challenge that we face is going to be like, how do we untangle these huge databases and monolithic apps that are, you know, years old and, and get those into like product-based microservices, right? Questions like, where do we host them? Are we going to host them on-prem or in public cloud? You know, picture a couple hundred development teams at State Farm immersed in the platforms of, uh, which the apps they manage reside and gaining a lot of expertise in each one of those platforms and really feeling like maybe they haven't had the chance to branch out and learn about some of the newer tech and the more strategic environments that, that we're asking them to go to. So with those teams, like my, my team assist those teams by working with them. And we learn by doing through what we call our cloud native dojo. So in that, the dojo kind of setting, we, we invite teams into a big collab space. We coach them how to get to one of these

4:10

cloud platforms with continuous delivery and in DevOps principles, but we can't do it alone. So we have partnerships with other teams like, you know, learning and development is, you know, a team that at State Farm that allows access to different learning platforms and media. We have a public cloud adoption team that has, it gets access to the public cloud to our teams and provides some curves and, and things like that for, for people to deploy to public cloud. We have an SRE team, site reliability engineering team that will help, you know, developers understand when they create, you know, infrastructure in a self-service way, if they're on or off prem, you know, how to build in resiliency and availability. And then one of the key teams that we work with is, is a team called migration services. So this team is a team focused on gathering up like all the applications that are targeted for modernization that might be sitting on a, one of the legacy platforms. And they kind of wrangle the cats to make sure that the work of modernizing and getting to the more strategic target platform, seeing that that work gets done. And again, we have the most tight partnership with that team because they help us like identify the teams that we can help the most and then prioritize those. And there's other specialized teams as well, but like, those are the key ones. Yeah. All right. So this is actually, this is a topic we've been talking about quite a bit on the podcast lately, which is like migrating workloads to new platforms. And I'm curious what the, like when, when you, or when you're, when one of your teams is looking at a new platform or whether it's, you know, like you said, the PaaS platform or Kubernetes or the public cloud or, or what have you, or, or TAS or something like that. Well, I mean, I've got a couple of questions. Like one is who's trying to, who's driving that decision. Is that the developers themselves or is it the, is it kind of, you know, based on the applications needs, I guess the business needs. And then also just like, how do you, how are you making those

6:12

decisions? I mean, how do you, how do you decide or how do you prioritize one platform over another? Yeah. Well, you know, we've got many applications, right? Like we have one of our more legacy platforms has about, oh, a thousand services and apps that run on it. Right. So we have, we have quite a lot and, you know, work workload placement is, it's a big deal. You know, it's, it's using all of the, the, you know, information about not only how the application runs now, but how we'd like it to run to make those decisions about, you know, where to, to host where the ideal place to host those applications are like, but you know, I, I'm really proud of our state farm leadership team promoting growth initiatives and getting our dev teams access to so many materials for learning and time to do that. So I'll say, you know, we have a lot of applications, but that I would say the applications themselves and how they'll be kind of properly carved up into product-based services dictates where they'll be hosted. And I, you know, I, it's because like, I feel like we have really strong development teams that are capable of learning and that we have a lot of teams like mind, like my team that will assist in, in coaching them. So I'll say the applications themselves, you know, we'll put apps where they belong and upskill where we need to. And when it comes time to that, you know, upskill process, what are some of the considerations that you're looking at when you're coaching developers or helping dev teams onboard to a new platform or adopt a new technology? Because, you know, I think that really does make sense putting the needs of the application first, but there are, I imagine a lot of people and process considerations that go along with those applications when you're moving them. Oh yeah, absolutely. So I think one of the most critical things really, you know, when you come up with a, maybe a new target platform or you want to move applications off of a legacy

8:18

system, it, the most important thing is just to know the background of the developers who you're asking to, to make these moves. You know, again, we're a big enterprise with some choices on hosting platforms. We have a few, a few platforms versus it being completely wild west. Right. And that has afforded us the ability to know the experience and skillset of our developers. So when we sunset technologies, you know, we know what our developers are used to because, you know, they're, they've been consumers of a specific platform for a while. And then you ask, you have to ask yourself like, so how similar is the new platform or technology to what they've experienced before? And really at that point, you have the ability to kind of understand and tailor the coaching and the onboarding to those development teams. You know, it's, it's analogous to the old saying, you know, know your customer and you're going to be successful, you know, know your development teams and the platforms that they're moving from too. Right. And you set a, you set a date goal, you know, support from the highest levels of leadership, provide the funding, track the progress and hold people accountable. And, and it will happen. Could you go a little bit more into detail about this cloud native dojo program you have and, and kind of what this coaching looks like and what kind of methods or activities that you find most effective? Oh yeah. So it really, it depends on the objective, you know, what hosting environment are the, the teams targeting that we're, we're helping coach? Are they targeting just a platform as a service or are they targeting like public cloud managed services? And so like, you can think about like with a PaaS, you have like a tightly constrained number of technologies. You can get, you can get by with having a well-organized Git repo with lots of examples for them to copy an app and kind of experiment with. We've also created, you know, online labs that walk developers through deploying an example and getting it to work. But with, with public cloud, there's, there's just many technologies,

10:20

right? Which is great. You know, you got machine learning, AI, all the things, but it's less feasible for a team like mine to create examples and labs about all those things. So rather we teach disciplines over specific technologies. We instill like continuous delivery and DevOps mindsets so that people are able to quickly deploy, learn, learn by doing and, and fail fast. So really with public teaching disciplines is key, you know, and I would give example of what I, what I mean by discipline, like automating everything through continuous delivery pipelines, using declarative infrastructure as code tech, automated testing and security scanning. We use chaos engineering to learn resiliency, the chaos engineering tools. And then, you know, using a system that really streamlines production changes where developers can deploy to production and in minutes instead of days or weeks. You know, there's a book I, I really like that DevOps handbook, you know, and there's a quote from there that I, I repeat quite a bit from the DevOps handbook. It's the, the faster that we can experiment, iterate, iterate and integrate feedback into our product or service, the faster that we can learn and out experiment the competition. So again, we've got rich data from, you know, a hundred years of doing business. So, but so does our competition, you know, being agile as possible and getting our developers, the ability to very quickly experiment and learn gets, gets more critical every day that new techs made available, not just us, but to our competition too. And we need to out experiment them. Right. I want to back up, like rewind a bit, Jeff, and ask, because you're talking about like teaching developers, but how to, how to get acclimated to new platforms. But when you were talking about your history that spans back a long time, I'm curious, I'm curious what the learning experience was for you as you went from like a cobalt back in the day to, because I just, every time I hear cobalt, I think about the unemployment systems when the pandemic started and they were trying to find these cobalt developers from, from decades ago, who were the only ones who could fix these systems.

12:22

Like, like what was your experience trying, getting up to speed with this stuff? Because the learning curve is not, I mean, it can be a sharp learning curve. Oh, absolutely. Yeah. And I, I, I could probably brush off my cobalt skills and help out the government, but I, you know, so it's, that's an interesting question for me, you know, it's been quite an interesting path. And I think the one, the path that I've been on has been kind of fortuitous in the sense that, you know, my first experience after the cobalt stuff at the green screen stuff in, in managing, you know, our, our first paths or not necessarily paths, but our, our first web hosting platform, you know, that was one where, you know, I, I think it really helped prepare me for the environment that we're in today in the industry. So, uh, example of that is, you know, I had to maintain install is webs for application server on, uh, Linux, you know, x86 commodity. And it was kind of a shared hosting environment. So I had to, you know, make sure that it performed, you know, had to make sure that it was resilient and really all the things that developers have to do now when they target, you know, a cloud system, you know, as everything's shifting to the left, people have to be responsible for all that stuff. Well, I was responsible for all that stuff for everybody for a while as an administrator of the, you know, our, our web hosting platform. And, you know, I had probably 10 years, 10, 12 years experience doing that. And so I got very well versed in, you know, why it's important to do your due diligence with the performance testing and make sure that you try to do run a little chaos, you know, before chaos engineering was a thing, you know, where I'd kill processes and drop nodes from a system and just make sure that everything still worked, you know? So, so I was in the business of doing that for a while, you know, where our application

14:25

development teams used that system, you know, we had tools, it evolved into a custom like PaaS cloud system where, you know, people could self-service, you know, deploy their applications, but they, you know, we did a lot of stuff for them, right? They didn't have to understand, you know, what would happen if I dropped a node because that, you know, I, I kind of did that on their behalf. But nowadays, you know, again, with the shift to the left, you know, the industry is trying to get out of the way of developers and let them do everything, you know, create your infrastructure, you know, do resiliency right or wrong, maybe at your own peril. So now I'm in a spot where I'm coaching development teams to kind of understand, you know, what my thought process was when I built those web hosting platforms in the past. And I think having that context really helps, helps me to, you know, coach teams properly. And I feel kind of lucky to have the, had the journey that I've, I've had at State Farm, because right now I really enjoy my job and, and I have a great team. And I think that we're able to really effectively help development teams. Do you find that there's any, do you find any friction between, you know, teams needing to move further to the left? Like, are there folks who, who really don't want to do that? And, and they just want to stick to something that has a little bit, you know, more of a PaaS experience? Or do you find that most teams are kind of excited and interested in learning those skills? Well, you know, we, we've got a lot of developers. I, I would say there's, there's pockets of developers who were really charged up by the prospect of, of shifting to the left and doing more and understanding more because, you know, it has been their passion. And, you know, when they go to work, they do it at

16:25

work, but then they go home and they do more at home for personal projects and stuff like that. And those, those folks definitely get charged up about knowing more and doing more and having access to all this stuff. We do have other developers that I would say aren't maybe as charged up, but they, they're good at their jobs and they understand the importance of, you know, shifting to the left and learning everything. It's just, you will get some feedback. That's like, wow, this is, this is kind of a lot more to know than what I was anticipating. And, and I think that's why, you know, teams like mine exist and, and we do a good job in trying to, you know, kind of spoon feed a little at a time for, for teams so that they don't get completely inundated by everything that they have to do. And, and I, I do think that, you know, we do a great job at State Farm with, you know, not only learning materials and providing a lot of access to, to different online tools and things for people to learn, but we also have a lot of great teams that are, you know, specialized that'll help teams, you know, with whatever, you know, they need help with, you know, from us doing cloud native dojo to other teams that, that coach how to do continuous delivery and pipelines and be agile to, you know, we, like I said, we have SRE teams, we've got performance teams, and these are all, you know, teams that can help with like consulting with, with questions or issues that, that anybody's having. So, you know, it, it's, it is a lot, the landscape of IT today is it, there's a lot for developers to, to learn now, especially as, you know, these cloud tools give them access to, to really do everything, but we, we have good support in place for them. Do you find that that leads to, I mean, cause you, you mentioned SRE a couple of times and, and one of the big things we talk about SRE is reducing toil on, on the people involved in the system. You talk about, you know, people often talk to what developers want to be challenged. They want to have access to, to new tools and stuff. I'm curious, do you find that these, these transformations have,

18:29

have made for happier, happier operations teams and happier developers? Cause I mean, I think that sometimes it gets overlooked. It's like the tech is one part, but you also need the people engaged and willing and retained in order to actually do it. So. So. Oh yeah, absolutely. You know, one thing that I'll say is absolutely, has been absolutely huge for us. You know, we moved a fair amount of applications to what's Tanzu application service now. And when we did it, you know, we had a goal to move. It was, I think 800 applications that we got moved in about a year and a half. And when, you know, when you do those kinds of migrations, you're kind of like, okay, how much can we ask for development teams to, to change and to maybe adopt what kind of things do you mandate? You want to be careful because all the extra stuff that you add, you can maybe jeopardize your date goals. But one of those things was mandating automated software change deployments. So what basically application teams have, you know, pipelines that they can run where, you know, they can deploy a change. Like I said, they can deploy a change in minutes instead of, instead of what it used to be like, you know, weeks or days or weeks. And I think, you know, I think that has been one of the biggest things that has increased developers happiness, because, you know, I think the more process and the more time that you kind of add between a developer, you know, testing and proving that their, their code is, is working to get it to production. I think if you can provide system and support them and getting that done super fast, that's just, that's a huge thing. And, you know, it, I would say that that's critical also in learning, you know, as you know, we really encourage learn, learning by doing. So, you know, whether it's like a system like Tanzu or Kubernetes or public cloud, you know, having, you know, those,

20:34

those, that automation and the pipeline and the speed to get their applications to, to test and then eventually to production, to be able to rapidly, you know, prove how application features and technologies work or don't work, you know, failing fast. I think, you know, that has been one of the biggest things to really help increase happiness with our developers. Yeah, that makes a lot of sense. Having those really fast feedback loops. It's really rewarding. Makes your job easier. We have someone wrote a blog post on that for us a while ago, about mentioning like the dopamine hit, right? And like the more you've accomplished, every, every little accomplishment feeds you, right? Rather than that slog of like, start over and back, back at it for another three months of grinding on this thing. Like I think, yeah, there's a... Yeah, absolutely. Yeah. Yeah, that's great. You touched on this a little bit earlier, just talking about how there's, you know, when you're training folks to work on public cloud, there's just so much technology to learn that you really focus a bit more on certain skills and disciplines. And I wonder, you know, kind of looking overall at all your tech stacks and all the different platforms that you're coaching developers on. Do you think it's those disciplines that is a little bit more challenging for folks to adopt or is it, is it the tech related stuff that's more challenging these days? You know, I, I would say, you know, process related stuff, you know, as far as, you know, developers getting access to things, stuff like that. I mean, that can always be a danger, you know, of slowing people down and slowing teams down. And it's always been a challenge for, you know, large risk averse companies like us, you know, but again, I'm, I'm proud of the progress that we've made recently, you know, with, like I said, with moving around like the 800 apps to the

22:34

TAS platform that I, I mentioned, we, you know, we mandated use of the, the automated pipelines and the automated chain systems. So, you know, we, we got a lot of progress made there from, from the process perspective, you know, with, with the adopting technology, you know, the, the industry, again, has just had this intense focus on giving developers self-service and, you know, they just need to understand all these things, right. You know, infrastructure, they need to understand resiliency, you know, again, you know, like the past systems of the past, the kinds that I administered, they take, they took care of a lot of that stuff for them before. But, you know, shifting to the left again, that's, I think that's going to be like the biggest challenge for developers really in the, in the tech realm is just getting a good grasp over all that stuff. And I, again, I feel fortuitous in the sense that I've had that experience, you know, and, and I've, you know, maintained the web hosting system for a lot of people. And I've, I've built up all these disciplines, you know, that, that have become important in doing due diligence, to make sure that things work. And I, in part of my experience, I, and I'm, you know, I feel like if I can do something, anybody can do it. So it, with that experience of the web hosting platform, you know, I built up a lot of the skills and the disciplines over a number of years, you know, I was in that spot for like 10 years. So I, I do sometimes, you know, when I'm kind of coaching a developer, I feel like, Oh yeah, I did this. You can do this too. And it should be pretty easy, but I kind of take for granted the amount of time I spent in learning all that stuff. So it does take time. And, you know, I think once developers, especially in the industry now, it's, it's, it's like, it's really going this direction where everybody is going to really need to shift to the left. I think once developers get that, that, that mindset and they practice it for, for, for a while, they'll start to understand the disciplines more and really, you know,

24:38

they'll, I think it'll turn out to be where the disciplines are the things that are more important than any specific piece of technology or managed service. Right. Cause we can always, you know, learn pretty quickly about maybe a new language or, or how a managed service works, but really having the, the disciplines instilled in you to know the best way to, to, you know, research and experiment and make something work properly. I think that's, that's going to be the big thing for people going forward. I want to ask too, just, you know, referencing some of those different environments, like, like, like Kubernetes versus a past platform versus a public cloud. Like, I'm curious, I'm curious specifically on the public cloud angle, like how you differentiate or distinguish between, you know, running on the public cloud versus running on Kubernetes, for example. I mean, do you, do you run, would you run Kubernetes on the public cloud or is it kind of a, or is it when you talk, when you say public cloud, do you mean basically public cloud managed services? I'm just curious, I'm always curious what that distinction is with companies. Cause I, cause sometimes you hear it and it's like, I don't know that they're mutually exclusive, but maybe even the circumstances they are. Oh yeah. You know, I, there's so many options really with the public and, and when I say public, I, I, I mean, you know, in regards to really anything that you can do in public cloud. And that includes, you know, installing, you know, Kubernetes cluster and running things on that, or even installing a paths on infrastructure that's in public, you know, I mean, that's an option too, right? So there's a lot of different ways you can slice it. We've got Kubernetes clusters on-prem, you know, we've got paths on-prem and I, you know, I think some of those choices, you know, you look at the skill sets of the development teams, you look at the complexity and the makeup of the application to see if what, what system would be better suited for. And then, you know, if there's a desire to get to say public cloud, there's other things that you have to really

26:40

consider with, you know, we have a lot of high transaction systems and services, you know, on different platforms that have to be integrated with one another. You know, if we have, you know, all these services on-prem in our data center, is it going to be okay to have, you know, one of those services move to say a public cloud and then maybe, you know, what does the latency look like? If it's a high transaction system, you know, is it going to be acceptable to take on a little bit more latency, you know, or, you know, and then, you know, questions like how do you carve up the data and shift that to public, you know, or is there a bunch of data on-prem? So yeah, we've got more choices now than ever. And it's going to be fun as we, you know, modernize, as we, you know, kind of decide how we choose what platform any given service should land on. Right. I wanted to ask too, just following up on an earlier conversation about developers shifting left, which is what you, if you get a sense, like, well, if you have a sense of what the ideal abstraction is, as, as developers are shifting left, it's kind of like on the one hand, they're taking on more and they're expected to understand more. On the other hand, you want to make it as easy as possible to do the right thing. So I'm just curious what that, what that ideal abstraction layer might look like. Yeah. You know, it, you, ideally, you know, development team would just kind of run with all that stuff and be good at all the stuff, but realistically it's, that's probably not going to happen. Right. So that's, that's where we have these, these develop, these peripheral teams that really kind of help with, with any given topic really. And yeah, I do think that's a challenge. And in regards to, you know, how difficult it's, it's going to be to, you know, shift to the left, but you also look at our, our product teams and how big they are. You know, I think we've done a

28:43

pretty good job at the, having the ideal kind of product team, you know, two pizza team for a lot of our development teams and what you kind of really hope for with a lot of those teams is a good, diverse set of people. Right. So you might have somebody who's, who's really knowledgeable, more knowledgeable about the infrastructure and one that has a little bit more background and say, you know, you know, testing and then some really strong developers and they kind of, they collaborate and kind of help each other out and hopefully kind of cross train, you know, each other in those disciplines. So I think, you know, to get to the, the right spot, maybe most efficiently is to have that like diverse team. I think you kind of answered my next question, which was, you know, looking ahead next five years, like what disciplines would you really recommend developers hone in on for their career development? But it sounds like, you know, that diversity is really key and it takes, you know, a whole bunch of skills. Yeah, absolutely. You know, I I'd say, you know, knowing at least a decent amount of all the disciplines and then, you know, getting help when you need to dive a little bit deeper is probably the most feasible approach. And then, and then finally, Jeff, maybe we can just end, you know, on another, on another, let's call it a prognostication question, but like, when you look, I mean, because this can be State Farm, this can just be industry-wide, but I mean, where do you see, like, I mean, I'm curious what, what technologies and maybe even what platforms you see having the biggest or process that having the biggest impact going forward or where, you know, you think, okay, like we're, if we're investing heavily in one place over the next five years or 10 years, it's, it's going to be here. I'm curious what that, what that might be. Well, you know, I'd say, you know, as big as like the public cloud providers are

30:46

now, I think they're only going to get bigger, you know, as you know, there's a lot of like, you know, tech companies, obviously that I'm already doing a business out in public, big companies, you know, many companies. But, you know, I think as some of these, these really bigger, larger enterprises, you know, having, you know, outages in private data centers, you know, that kind of stuff's going to happen, you know, over time. And as some of these bigger companies have those experiences, you know, they're going to look to some of the, the huge cloud providers, right? You know, so even, you know, very large enterprises, they might have only like one to one, two or three physical data centers where, you know, public providers have dozens of, of data centers, right? It's an easy math problem, you know? So I think as big as those cloud providers are there, they're probably only going to get bigger over time. And, you know, I think once, you know, big companies with a lot of data, you start to get out to public, you know, there's going to be a, another push to really understand and get the value from the data with tools like, you know, artificial intelligence and machine learning. So that's, that's kind of where I think that the trend is going to go in IT. It's, it's probably kind of an easy guess or statement to make, but I think, you know, as big as it is now, it's just going to get bigger. Well, Jeff, it's been so great to have you. Thank you for sharing all of these interesting insights you have and talking to us about what you're doing at State Farm. You're doing a lot of really forward looking things there. And it's, it's been great to learn about. Oh, thanks so much for having me on as it was fun. Thank you.

Tanzu Talk · 🍕 Get more pizza.