Not standing still in Secure DevOps with Jai Schniepp of Liberty Mutual

Episode 123 · 23:46 · SRT · archive.org · all Tanzu Talk

Hey listeners, thanks for tuning in to another episode of Pivotal Conversations. This week we sit down with Jay Schneip, Senior Product Owner of Secure DevOps Platforms at Liberty Mutual. And what I really learned from this episode was the importance of you can't stand still. When we talk to Jay, you'll hear again and again how her team has relentlessly focused on reimagining the solutions that they build for developers internally at Liberty Mutual. So enjoy. Okay. All right. So here we are at CF Summit, Philadelphia. I'm with Jay Schneip. Schneip. Schneip. Okay. From taking German for that one year in college, like the EIs, now it's all squirrely in my head. And if it's not right in front of me, then it's over. Okay. Schneip of Liberty Mutual, who's been on main stage and gave a breakout. So wanted to kind of follow up and just hear more of the Liberty Mutual story from your side of things, Director of Product for Secure DevOps, which I kind of love that like that's woven right into the role definition. Yeah. What better way to say security is important than to put it in your name? Yeah. Like, no, really, that's what we mean. Yeah. And I know from, you know, hearing you speak before and our conversations that that is actually really important to the way Liberty Mutual has kind of built up their platform around PCF. But let's start at the beginning. Sure. So where do things kind of start for Liberty Mutual and then we'll take it from there. Yeah, sure. So we started our journey in 2013 when we, we being a very small team of maybe six to eight developers and myself had a need for an easy to use platform. We were building web services

and we had the opportunity or the only option, if you will, was to leverage WAS and the WAS deployment and implementation would take eight weeks per environment. And we were AJ, which meant it was times two and then per environment, dev test prod. It was a long time and it seemed really heavy for an implementation like ourselves, where we were simply trying to provide web services out to the business units. So we brought Cloud Foundry in and we implemented it. And from the get-go, like you mentioned, security was important. So we immediately abstracted the CLI and we developed a front end where users could start and stop, connect AIs and applications from that portal. For our team, it was great because we were the only people using it. We could, you know, manage it in a way that there was some limited access in places. As other teams and organizations started to be interested in leveraging it, we had to put a rollback model around it because we needed that granular layer of security. Because everything we do, we're really focused on least privilege. Okay. And so, you know, starting out, talk about the sort of adoption trend in terms of the number of teams. Like how did that, well, maybe if it's kind of a long time ago now, but, you know, even just the shape of the curve, right? Like how folks were onboarding starting with the, you know, roughly under what time span did the pace of the number of teams you had to support in this model kind of grow? Yeah. It was, I mean, it was certainly slow and steady for a while. Small teams, small organizations looking for a fit for purpose solution like this. It really wasn't until we signed on with Pivotal

that it really started to escalate. And really because we knew a lot better of how we were going to handle the load. So when we first started with Pivotal, we, we also were very opinionated about how you get into Cloud Foundry. So we were asking teams to leverage pipelines, which were still relatively new from a Liberty Mutual perspective. And teams just didn't know how to get started. So we partnered with Pivotal on that and we did a dojo where three members of the team went down to Boston for, I think they were there, six to eight weeks and really started to ask the questions, do the user research, talk to the teams that are interested in Cloud Foundry already, but also teams that weren't leveraging yet. What was the blocker? Why hadn't you started? And from that relationship and from that dojo, we built the first real console for starting at cloud, for starting in Cloud Foundry that we called Pipeline Provisioner. Okay. And what that did is provide you with the initial starting template to get into Cloud Foundry. It included your build and deploy pipeline. So is this like, there's a set of deployment pipelines that are already kind of pre, pre-cooked, if you will? The templates are cooked. So each team still needs their own because they're in their own organization. So we can manage the roles. But yeah, so there's already a pattern there for teams to consume. So what we did is create those patterns that were easily consumable for teams. Okay. So you kind of get a, yeah, a pattern to start with, and then you have to sort of add in a little bit of the, you know, you add your sprinkles to the sundae, the sundae being the scoop of ice cream and the essentials, which all, you know, have to include chocolate and cream. Obviously I'm opinionated about that. And the cherry. And the cherry, but you know, whatever. So we digress. Yes. Slight, slight side topic. Important. Right. But maybe not for today. So anyway, so, okay, this,

this console pipeline folks can then log in and, and, and what? They sort of are able to pick kind from a menu of pipelines where you've already baked in the security side of things. So not yet. So they were able to request a build and deploy pipeline that got you into Cloud Foundry. And we, you know, after we deployed that, after we deliver that, we're high-fiving. We're like, look at what we did. Isn't this amazing? So smart. User research got us here. Like we're awesome. And then teams used it for about a year. Okay. And then we found it wasn't enough. They wanted all the things. So what was missing? What was missing? It was really the kind of the ability to go end to end. So it had some of the things that they needed, but not all of it. It lacked the holistic approach and also provided a disjointed experience. So as teams were leveraging the pipeline provisioner, it didn't tie into DNS and it didn't tie into Liberty Enterprise Identity. So it kind of just got them started. So then in 2018, well, really the end of 2017, we started to do more research. Okay. And we started to say, what thematic problems are developers facing right now that we can start to tackle for them. Like, why are they saying this isn't enough? Okay. And what we found was we needed to increase the knowledge of our opinions. So why was a pipeline required? You know, what were the basic principles of consuming cloud and making that available for teams? We needed to even increase awareness of what options were available. Okay. So it was almost like, wow, we know what we can do, but we weren't doing a really good job communicating to others what was possible. Okay. Yeah. So that's interesting. And a lot of dimensions, right? This is like, just the notion

of kind of revisiting something that was sort of seen as a solved problem. Yes. Um, and you know, we're not accepting, uh, like the, the kind of the status quo, right? Like that, I think that's a really healthy approach that, um, embodies what we sort of often talk about in the kind of, you know, you're building a platform that's a product in some ways, I think maybe offering would be a better term, right? It's, you know, it's more of a service, but again, uh, could digress into the semantics there. Try to avoid that right now. So you're, you're building this offering for your developers and just like, you know, any, any enterprise for their end users. Yeah. Right. You can't stand still. You can't stand still. You can't wait. You can't assume you've done it all. You know, and I think what's really, you know, interesting when you think about a platform engineer or someone that's supporting the platform of Pivot Cloud Foundry or supporting even Docker containers or AWS, you can't be so audacious to think that, you know, what your developers are experiencing because you have different set of privileges. You have a different set of experiences. You've, you know, probably in most cases been in the platform longer and you don't know anything about the applications that they're trying to deploy. You know, we don't have the, you know, we're not fortunate enough to be building new, new applications every single day. You know, we're, we're not doing greenfield work. We're talking about applications that have been, you know, long running applications at Liberty Mutual and we either need to replatform them or refactor them in a way. So they're going to work, you know, in this environment. So we have to ask the questions, we have to get digged down into what are the blockers? What problems are you trying to solve? What are you experiencing here? So then we can start to figure out what those solutions are. And how much of it is also like, if we think about, you know, maybe that first pipeline provision or

tool, I'm guessing maybe in the 2015-ish? Yeah. Yeah. End of 2016, early 2016. Yeah. Yeah. Okay. So from there to then late 2017, I mean, just like with all things in technology, you can, you can pick whichever timeframe you want, but nonetheless, when you have almost two years go by, like, you know, the market doesn't stay still in terms of what's available to developers and undoubtedly, you know, the kinds of patterns they're trying to adopt. Did you see any of that factoring in as well in terms of the demands that those developers had where they were not being met? Or was it more of just the business side of it is of, you know, in order for me to do my job, like, forget the technology options that, you know, might now be available to me. At the end of the day, this is, this is just a process issue, uh, that I just want to be better. Yeah. But it just, I'm curious, like, you know, 2016, 2017 was obviously like a lot of stuff was changing in the landscape, you know, Docker, Kubernetes, CICD tools coming out of the, out of the woodwork. Um, how much of that played in, played into this? I think, I think a lot, but then also maybe not so much. So it was really starting to understand what was blocking people at Liberty. So as we got those easy, I'll call them the easy applications into Cloud Foundry, the ones that, that fit very well, they were either already trial 12 factor or, you know, had the majority of the factors, um, and they were ready to go. We started to look at, um, applications that were going to be more difficult. They weren't going to be easy wins. And the blockers that we started to talk about were identity, right? Again, so, you know, integrating into our enterprise identity solutions, having vanity URLs incorporated into the pipeline itself so that when you do push to production, everything's ready to go. Um, starting to think about secrets management, um, and how we want to

manage that from an enterprise perspective. So really that next step for us after pipeline provisioner was getting all of those things incorporated into a platform and making it consistent. Cause they also came to us and said, we really need, um, to unify what we're doing. It's hard for teams to, to start with a pipeline and then they have to go talk to networking to get, um, you know, what they need from DNS. And then they need to go talk to a completely different organization to get an identity and then another organization to get a secret and really starting to look at, you know, liberal liberties, IT organization as a holistic structure and say, we can deliver all of this for people. Why are we making them go to seven different spots on the way to production? Okay. So then, um, the other, the other thing that you mentioned was around awareness of their options, um, which, you know, is in my opinion, a different challenge, but also plays into the same notion of you know, this is your, this is your product, this is your offering and, uh, you, you, you need to market it. Right. And the marketing isn't always just like a, a, a pretty brand. Right. Uh, it's, it's a communication plan. Exactly. So, um, tell, tell us a little bit more about what happened in terms of changing, changing the awareness challenge. Yeah. So pipeline provisioner was almost primarily used for cloud foundry, um, which was great, but you know, teams were also starting to look at AWS at the time. They were starting, you know, to your point, looking at Docker containers, there was a lot going on at that time. So awareness from the fact of what options did they have available to them to consume and really starting to believe that the developer or, you know, the organization itself is going to make that decision. And it's not going to be someone like, we can certainly have our opinions and there's always going to be a more appropriate solution for what you're doing, but we certainly don't need to be the team. It shouldn't be the team telling you which solution solution to

consume. So as we started to pull together what we call the cloud forge console today, it was really about one that end to end experience. So DNS is there and ready to go, you know, identity is there and ready to go. Secrets is available, but then also, oh, you can do a cloud foundry instance here, or you can put something out on EC2 in AWS, you know, or do you need a cloud foundry and an S3 bucket for your storage? Great. Here are, you know, here's the way to go and starting to build those templates and those end to end pathways to get to production that included not just the platforms and tools that we manage and support, but the other ones at Liberty Mutual that are critical to building an application. Okay. Um, so yeah, those deployment targets and some variety there. Now, um, so are you sort of saying that you solve the awareness problem by building it into kind of the, the console tool itself, um, to sort of surface up these options? So we've, so we've, so we've, we even hate saying that we've solved anything yet, right? True. Um, I think we, we improved it greatly, but even just recently within the last three months, we've pulled together now a team that we call university or university plus. Our next step is to understand where is there missing knowledge? You know, we've been on cloud foundry for, you know, do the math six years. We still have teams calling us and saying, my AI is down and we're like all of them. Nope. Just one. Why do you care? You know? Um, and starting, you know, so there's, there's education there, there's education on which tools to use and when, you know, when I talk about the most appropriate place to deploy something, again, teams are always going to have that choice, but how do we educate them either from a financial perspective or a security perspective on, you know, where they should be putting things or again, should is even

probably the wrong word there. Um, you know, what options they have and what makes sense based on what they're trying to deploy. Um, so this, this whole university team is really important to us because it's one looking for ways to improve that, that end to end product and end to end services that we provide. But then also starting to understand what it really means when I say contextually relevant knowledge. Okay. So we're not looking to build a knowledge base. We're not necessarily looking to put out a bunch of wiki pages, although maybe we start there, but what experiments can we do as part of that team to say, do videos work? Do podcasts work? How do we, um, improve help text within the process itself to get people the knowledge that they need when they need it? Okay. So, uh, tell me a little bit more about this team. And so they're running experiments to sort of figure out how do we get the information into the, to the space between the ears? Um, you know, how big is the team? What are some of the things that they've tried? You know, would be, can you share any of the things that have been successful? Just other folks, I'm sure are facing the same challenge. Yeah. I think developer enablement is a really important topic. So I'm always curious of like little nuggets that people can share. Yeah. And I really do wish I had more like literally the team is probably, um, maybe three months old. We just got a product owner within the last month and a half. Um, but the, and the team is very small, you know, starting small, let's experiment, let's see what happens. And right now, um, they're in aligned and embedded with our support services team. So our customer success teams, you know, they have so many questions, so many people call in and it's for more than just something's broken. It's how do I do this? How do I do that? Um, you know, one example of something they're going after right now is when you look at our JIRA, um, implementation, there's 20,000 users in JIRA. We only have 5,000 Liberty

IT employees, which means there's over 15,000 non-technical users consuming that product. And so when you think about what it means to create an LDAP group, role-based access management, um, even identifying identities within the platform and understanding, you know, who goes where, that's something they've never experienced before. So how do we improve that experience for them? You know, and then you layer on top of it, all of the request mechanisms that Liberty has on, you know, you need to use these different services to request, you know, an LDAP group in development, then you have to do it in non-prod, then you have to do it in prod. And that whole process can be so overwhelming for a business user who's simply wanting to manage their work in JIRA. Yeah. Or they've partnered with an IT team and they need to, you know, set that up. So, so looking for, for nuggets like that, using data to drive which small wins we go after first. Um, so some videos had been done a couple of years ago, um, on helping teams with JIRA. We don't know if they're being used. So we're, you know, start, what, what do we have? What else can we try? Um, you know, and then just continuing to iterate from there, you know, what other small wins can we have, you know, that come from the customer success teams to say either, you know, is, is success here eliminating calls? Is success here, you know, improving the customer experience? Is success here keeping calls, but, but doing a like one call and done kind of perspective. So we have the documentation, so it can be a quick send off, you know, and then what kind of level of service are we really trying to provide here? Um, and you know, so there's a lot of questions, a lot of things to be answered, but it's really exciting to think about, you know, how do we get that contextually relevant knowledge to the right person at the right time? Yeah. Okay. Well, this seems like a good topic to

definitely follow up with you or someone on that team, um, in the future. Uh, cause I'm super curious. There's a lot of really great questions that you guys are clearly looking at. Um, and so I'm just curious as some of those answers perhaps start to emerge. Um, maybe someone will have something to say about it at SpringOne. Oh gosh, what an interesting setup, Jay. Um, uh, who knows? We'll, we'll see if the agenda is published, uh, how long after this podcast is published, but, um, okay. So just to kind of wrap things up, um, you know, one thing you mentioned on the main stage, uh, when asked, uh, you know, Hey, you know, fast forward five years, um, you know, do you, you imagine you're still going to be using cloud foundry? You, you had, you were pretty confident in your, in your yes answer. Um, a little, little tweak on that question is, you know, fast forward, even just two years, like, what do you want to see that your team has gotten done? Yeah. No, that's a good question. Um, I will, first of all, I want to see us out of the data centers with our PCF implementation. Um, we're working on that now. Um, we want teams to be running PCF and AWS. We want to, we want them to be running on that platform. Um, I'd love to see us be able to abstract some of the, the Liberty specific items we've had to put in place, like, you know, the abstraction of the CLI, you know, and have that role-based access control to a really granular level. And maybe there aren't other organizations that need that. So understandably, maybe it never comes into the open source space. Um, but I would love to, to see that and give, you know, teams a little bit more access to, um, really some of the real power within the platform itself. So see some of the abstraction kind of actually fade away away. Okay. And then also to follow up on the, on the point about you're already in the process of moving to AWS. Um, so is

that something that, what lessons do you have already from having, you started with PCF completely on-premises now you've, you've already started to run it in AWS. Any kind of lessons bubble up to the surface from you in terms of making that transition? I would say, just don't get behind. Don't get behind on your revs and, and re-platforming that or the, um, repaving that can be tough. Um, and, and we're really working hard to keep both instances, um, at least today in full parity. Yeah. Um, but I think, you know, we may, we may need to make a decision sooner than later that sure you can stay running internally, but you're not going to get, you know, what you can get out in the cloud, um, and start to just, you know, invest less in that space. Yeah. Okay. Um, okay. Well, thank you so much for joining us and, uh, yeah, look forward to seeing what, uh, what Liberty Mutual brings to Springland platform as well. Me too. All right. Thanks. Thank you.

Tanzu Talk · 🍕 Get more pizza.