Modernizing from Mainframe to .NET Core at Travelers Insurance

Episode 154 · 53:06 · SRT · archive.org · all Tanzu Talk

Okay. Welcome back, folks, for another episode of Pivotal Conversations. Today, I'm joined by Viraj Naik from Travelers Insurance and Rohit Kalapur, one of our app modernization experts here at Pivotal. Viraj, go ahead and why don't you introduce yourself? Hi. My name is Viraj, and I am a lead architect with Travelers Insurance, working in personal insurance line of business. I generally am involved in many applications, but recently I've been involved in cloud applications and specifically around cloud transformation, app transformation. So that's just me. Awesome. Rohit, how about you go ahead and introduce yourself? You've been on the podcast before. Yeah, this is my second time. My name is Rohit Kalapur. I am one of the app transformation practice leads at Pivotal Labs. What that means in simple English is I help customers migrate workloads and their applications onto Pivotal platforms so that they can get the value of the cloud and realize the outcomes that they want from application modernization. I'm really, I'm pretty hot to try on app modernization these days. And of course, in classic confirmation bias mode, I come across these reports from like Forrester that says application modernization is like the most important work you're going to do for the next 10 years. And Gartner has their recommendations out. So I'm constantly like sniffing out and getting all this feedback from analysts in the market that, yeah, this is really important. But I think it's all also challenging because it's not always, it's not always the sexy new stuff, but it's like the, just the critical work that needs to be done. So I'm really excited to have the two of you on because I think this is, this is just like the

stuff people need to hear about and know that it's, it's, it's not a no-fly zone. It's doable. Um, you know, here's how to do it successfully, et cetera. So I'm really excited. Um, and in particular this, this, this kind of, uh, uh, topic came about because, um, we've got an upcoming webinar, uh, with the two of you and it's focused on modernizing a .NET application off of mainframe. Um, and so that's another, like two other areas, two other kind of keywords popping up on my radar. Um, so we look at, we look at mainframe modernization, Rohit, you and I did a webinar earlier this year on this topic. And, um, it's sort of, you know, as, as much as people have said, you know, mainframe is dead, right? Like these rumors of the death of the mainframe have been greatly exaggerated and it's still around. And so, um, you know, there's been a long time when folks have, you know, Hey, we want to look at how do we, how do we get off of mainframes, et cetera. And yet that, that story is not new, right? Like you've, you've heard this for years. I've heard this for years, decades. So I'm curious and Viraj, maybe this is really more of a question for you. I'm curious, like what kind of was the catalyst that, that really made it happen this time? Because I'm sure that this was not the first time that travelers are like any company. It's not the first time folks have looked at, okay, no, for serious this time, we're going to, we're going to modernize off this mainframe. What happened? What was different this time around? Um, multiple things, right? Like I think recently we see that there are a lot of market disruptors, like there are a lot of things that forces, uh, change in strategy, change in direction. Uh,

so our business also is not, uh, insulated from those things. So they are also looking at ways that we can adapt to the market. Um, we also want to grow in the market space. Uh, and, and sometimes, you know, you just want to try out things like you, you want a quick experimentation and then also a quick implementation. And, and in those cases, you also want to make sure that you, you can scale, you can expand, you can contract, uh, you know, uh, you don't want to build something and just leave it because you don't know if that, that is the model that we would want to continue for next two years or six months or whatever. So, so there is that flexibility that, uh, we wanted. So what you described about mainframe is true that has not changed. It's still, you know, uh, alive taking, in fact, performing really well, but that's the part like flexibility, scaling, you know, in, in terms of, uh, going up and down in terms of capacity, uh, you know, that was something that we said we have to have, uh, we cannot really always afford to have like six months of planning or, you know, uh, having infrastructure, uh, you know, ready, uh, for, for these things or growth opportunities and opportunities seldom wait, right? So you just have to be ready. Uh, so, so that was one of the key drivers. The other was like, as we all know, the mainframe has been around, uh, for a long time, uh, the skillset, I mean, in, in terms of the resources that you have, who can work on it, who can maintain it, who can develop it are very difficult to find. It's very hard to, you know, kind of, uh, get hold of that. And if you had a mainframe based organization, you don't really, uh, you know, can ingest new talent and, uh, you know, like no one really wants to come and say, yeah, interesting mainframe. I want to work on it. Right. So it's just, uh, it was hard. So that was, and what that leads to is like, okay, it's total cost of ownership is also going up. So you have this flexibility issue, you have this, uh, you know, talent issue, and then you also have, uh, the cost

that goes up. So for every business opportunity that you see, you have to weigh in against, oh, do I have resources? Do I have, uh, you know, do I have infrastructure? Uh, so we said, we can't wait. We had done a lot of POCs, uh, you know, and, and we said, no, this has to happen as we grow, as we move along, this has to get at risk. We want to make sure that, uh, we have a distributed platform, which is not just distributed onto.net, uh, you know, technology standpoint, but from an infrastructure, uh, you know, standpoint also, we wanted to go to a scalable infrastructure that can go on and scale up, scale down and it is disposable. So tomorrow, if we say, Hey, you know what, this doesn't look good, get rid of it. That's fine. It's easy. It's so it's, it's just a marriage of convenience. So, you know, so as to say, we business was of the operation and looking for the opportunities, uh, we had this, uh, we had this and then we had a, like a cloud platform that, um, we thought is, is, is a good choice for us to make it happen. So that was different this time. Okay. I mean, you would, you mentioned some things in terms of, you know, wanting to be able to try things out, quick experimentation, quick implementation, which definitely that sounds like the antithesis of mainframe and like how anything gets done on the mainframe. So, uh, just to make sure I'm understanding this, you're saying that this, that was a blocker to being able to do that kind of quick experimentation and implementation. Yes. Yeah, you're right. Yeah. And that, that was one of the things that, that would slow you down is to kind of say, Hey, yes, we can, we can experiment on that. That's not, not a problem, but you know, to experiment and to keep the lights on. I mean, you have systems, you have enhancements that are ongoing, and then you want to do this experiment. Do I have resources? Uh, do I have the infrastructure to kind of say, okay, this experimentation would take, you know, this many amount of this much of MIPS, have I planned for it? Uh, you know,

can I accept, uh, can I handle load if I see that experiment, uh, succeed, you know, succeed and, and so on and so forth. So that, that always was the challenge. And yeah. Yeah, no, that's, that's interesting. Like the sort of questions around say, you know, capacity planning and capacity management with something that you don't know if it's going to work or not. So that's a, that's a, almost a business level hindrance. Rohit, does this sort of jive with what you see? Cause you've, you worked with lots and lots of customers, um, some modernizing off of mainframe, some just modernizing off of other middleware stacks, but did this kind of constraints sort of jive with, with what you see, or is there anything else that you, you see kind of come up with, with other users? Yeah. The primary two drivers are cost and agility. And in case of mainframe modernization skills is like a third aspect, which adds itself. So cost agility for most, uh, legacy workloads or classic workloads, and then skills is an added component in mainframe modernization. Travelers is kind of lucky in that they were able to escape the gravitational pull of the mainframe, but a lot of the customers, um, are still stuck there because all of their critical transactional systems are in mainframe and they are not just in COBOL. They are in COBOL assembler. They interface with a lot of external systems. They leverage a lot of the secret sauce the mainframe provides in terms of resiliency. So I would say that we were kind of lucky in that this was a compute intensive workload, which was kind of, which was, I wouldn't say easy, but we were able to pull that out. Um, we had, I think initially we did a whole bunch of travelers did a bunch of POCs to validate that this is possible. And then, uh, I guess when the engagement formally began, uh, on a proper note,

then we were able to kind of leverage that work and then pull out these vertical slices of functionality. I like the way that you described that as the, the ability to escape the gravity, the gravitational pull of the mainframe. And, and if I'm understanding this correctly, you're sort of describing that, you know, not all mainframe workloads are created equal and some of them are going to be a lot easier to modernize than others. And looking at how many other dependencies on other systems, et cetera, are one of those maybe indicators that, that folks can look at. Am I getting that right? Or is there another way to think about, you know, okay, uh, if the, if the workload has this type of criteria, it's in a better position to look at, to modernize off the mainframe. Yeah. I mean, this, this space. So you are absolutely right. It depends on the workload and the criteria you use to evaluate the workload. So this is one where, you know, as Rohit said, this is important and this is critical. And it was, uh, somehow the most important. I mean, if I were to call it again, I'm not, uh, you know, I'm not able to describe it. Everything is important. I mean, in all the parts of that mainframe system are important. This is like a giant or a big monolith that we're talking about, but this is where, you know, this has impact on customers where they can come. Uh, they want to see like, what is the rate that travelers is going to offer? And we want more and more customers to come try, uh, or get a rate. And, and as a result, yes, this is very critical, very important, uh, function. And it is in, in that sense, if, if, if a part of it had to be modernized and scaled, this was one, there are other parts where, yeah, you know what, we don't need to scale that. That is not as much used or that's not, you know, uh, the things that get, so if you can write it or migrate it on, on a cloud platform, great. But if not, um, that's fine. There is nothing, you know, where you need to scale up, scale down, or, you know, there are seasonalities to it. Um, but this was one where you,

you had those variations. It was super critical, uh, and you wanted something which is robust. I mean, if this clicks or if the business, you know, decides to do certain things, and if they want to continue doing that, then we wanted to be, uh, stable enough to sustain the workloads. And I think what was cool was how we implemented it. I remember now where we, when a request for rating a policy came in, we sent it both to the new system that we modernized and the old mainframe system to compare and make sure that the rates that we get are equivalent across the two. And then once we had enough confidence, we switched off the, we, we switched off the processing on the mainframe so that, and then all of the traffic went to the newer distributed event based architecture, right? So because it was critical, we had to introduce that safeguard to make sure that we were not regressing any logic or making sure. And we, we functionally got the exact same age that we got on the older mainframe system. Yeah. Okay. Okay. And this is kind of actually perfect transition into my next question, which is about, you know, what is the workload or the application itself and what, what role does it play? And so this is the, like kind of a, for someone's getting a quote for insurance. There's a rate rating algorithm essentially that has to run. And then Rohit, what you kind of described where in the process of making this transition, one of the important kind of safeguards was these things are going to run in parallel for a while. We've, we've rebuilt it on a, a new platform, but it's still running on the mainframe and we can now run these in parallel and check to see if they're actually delivering the same results before we shut down the, the mainframe one. So, and then, um, but you kind of described that this was a particular opponent that needed to scale. And that was one of the drivers for identifying it as a good candidate to look at moving off of the

mainframe because it had this sort of, you know, I don't know if it was seasonal, but just variation in the scaling demands. So, uh, so kind of come coming back to, you know, the, the original catalysts, right. As a business trying to be more agile, um, what were some of the things that were going to place more, uh, demand on this particular application that you were anticipating these, these sort of scaling events were going to start to happen. I mean, um, one of the things that we really wanted to try out was, you know, kind of integrate a different, um, uh, analytical, you know, capabilities. So in order to do that, we wanted to see, Hey, if we stand up or we create, you know, this, um, rating, you know, engine, uh, how would it, uh, you know, kind of, we can do what if scenarios, right. We could, you know, run, we could do it, we could do it at a scale. So, so just to make sure that, okay, this is something that is working as Rohit described, you know, we want to make sure that the rate is accurate. Um, and again, as we kind of integrate, as we kind of, you know, say, okay, I want to tap into like, or when I say, I, the business wants to tap into a new channel, new market. Um, this was the part where, you know, what I am not going to plan for weeks, months and say, okay, do I have infrastructure ready? Do I have things? I'm just going to go and say, okay, just tap into that channel or, you know, kind of go into that, you know, uh, today we may have, you know, kind of had some artificial, uh, you know, limitations in terms of self-imposed so as to stay within our, uh, um, plans or budgets, uh, in terms of infrastructure. Now we can, you know, remove those, uh, you know, uh, blockers and say, okay, open. We want to do everything and anything that we really feel we should, we shouldn't have any,

you know, limitations, uh, impositions. Uh, and, and that was the, the thing that we, yes, we, we had to be ready. We had to kind of make sure no one said, Hey, this is the max that you, you could get, you could get anything. I mean, I mean, in terms of the workload or as long as you are accurate and we trust, um, that was the part, the scaling, uh, or the scaling. I mean, when I say scaling, it's not just upward, but you know, there are days when, okay, it's not, we don't have demands. Like on weekends, you know, typically you don't see as many. Right. We, we can scale down, um, or in some cases in worse scenario, if we see issues, we could easily turn it off and say, Hey, I, I don't want, um, to kind of use this, uh, you know, any, any more or tweak it, uh, that should be possible. Another thing that was very important that for the success of this program, which is basically has spanned multiple years now was the organizational commitment from the traveler side to right. Because in some sense, I mean, it, it got worse before it became better. We started migrating the workloads off the mainframe. We, I remember we created the distributed architecture. There were a whole bunch of microservices. There was some initial pain that we had to go through to land on the right, uh, set of services and the right target architecture, the right granularity of the services, uh, that will do all the processing. I mean, that's, uh, so it's, uh, in some ways you have to stick with it because the initially there is that pain in where it feels much worse and it feels like you're losing a lot of money, but then like the short term negative pain then, then results in a much more long-term benefit. So you definitely see some of that in mainstream modernization. Okay. That's, um, well, you know, first just to kind of close out on some of Viraj's points, it sounds like there really were kind of two factors with this specific application, uh, that were

sort of drivers around one being able to want to try out sort of different, you know, there's an analytical component to this particular application and wanting to be able to try out different things, uh, you know, just within the team that's operating that I think of it as like a, I'm sure it's like a service into other, other applications and endpoints. Folks want to be able to iterate there. And then the second thing was wanting to remove those artificial gatekeepers so that any of those other parts of the business that are like, Hey, we want to run this promotion, um, but not have to have weeks and weeks of capacity planning and, and process and bureaucracy to appropriately request the capacity to run that promotion. I can empathize with that. Right. As a marketing person, it's like, Hey, can we turn this on like tomorrow? Um, you know, like that's the, the, you rarely want the answer to be like, we need six weeks notice. Um, so cause all my best ideas are like shower ideas and I just immediately come in and be like, yes, let's do this right now. Um, I'm sure it works the same in insurance, like shower ideas prevail. Um, but yeah, you know, Rohit, your point here about, I think this is a great thing for folks to, to have on their radar and be prepared for, right. The whole, like, it's going to get worse before it gets better. And you talked about finding the right granularity of the services. Like, I think this is something that, um, I know, you know, talking to some of the Gartner analysts, they, they, they, they, of course got a bunch of different names for all these different things. Like, you know, I don't know, macro services or nano services or mini service. I don't know. They're like using all of the terms. Um, but I think the point that they're trying to make is like, you sometimes need these different granularities and you don't want to break things down too far because then you do introduce excessive overhead. So, but it's a bit of a journey to find that right granularity. Are there, are there any shortcuts that you think, um, exist out there

in terms of, you know, finding the right granularity? Is it better to just, you know, go really fine grained and then build back up? Or is it better to kind of go, you know, course to slightly less course and, and sort of get, get more fine grain from there? Or is it really something that everyone kind of has to figure out on their own? Because every workload is different. I think, uh, let Viraj answer part of this, but, uh, why don't you go first on this one? Yeah. I mean, I think I can, you know, tell you what we did was we started, I mean, again, we had two constraints, right? One, as Rohit said, um, this was something that is not a new business capability that, you know, we had a luxury of saying, okay, this is how it would be. It had to match in order to, so that this fits into the existing ecosystem, the output had to match, not just the output, but the response time also had to match, right? So all your decisions that you are making, uh, is like, no one is going to give this application, like, okay, I'll give you extra, you know, 500 milliseconds to produce your response. Uh, so you kind of have those constraints to start with. So you say, okay, yeah, no matter which I'm, and in this case we started with, like, I, I, I remember it was all, it was almost two years ago, but we had about 12 or 13, uh, you know, services because we were like true microservices, like, okay, only, you know, single function or single thing that it will do. And then we realized that, you know what, this is fine. It's, it's really good. You know, it looks good. Um, we are pure microservices, but then it had overheads like serializing, deserializing. And, and then when you look at it and say, okay, if I were to do it at scale, if I were to say I'm running hundreds and thousands of transactions, um, I'm going to see some degradation and can I, can I, I cannot afford to go over the response time that I have, or I have to meet that SLA. And, and that kind of drove us to kind of say, okay, can we consolidate, you know, some of those things? So, you know,

Rohit helped us with this domain driven design where we can say, okay, what is it that we can collapse? How can we not put, like, again, you know, create a monolith, but, um, just say, okay, these services that you have split apart, they can belong together. I think they, they have, you know, things that are common. They get, at least they are of the same domain. Can we put them together? And, you know, so we came down from 13 to, I think six that we have now. Yeah, we did. We remember we did that exercise of microservice rationalization. And what we did is we basically put every, each one of those 13 microservices through this six item checklist that Nate Chuta came up. Like he, there is an article that Nathan, should I put up the six factors of a microservice or some, I forget the exact title of that. But basically you, you take that, you take each microservice, you evaluate it on those six criteria. And if it stands on its own, it stands on its own. But if it doesn't stand on its own, you merge it with the other service on which it is dependent on. So we kind of use that six factor rubric to, to compress and compact as 13 microservices into six, six or seven odd services. The second thing I remember is that this was a very latency sensitive application, because when someone is really asking for a quote, he can't really, you can't really make him wait for so long. So there was a latency budget that really we had across all our individual microservices. We couldn't exceed that target SLA, which was, I don't remember, it was around like 400, 500, 500 milliseconds. But we definitely had this notion of there was a latency budget and each microservice kind of had a portion of that and all of them could never exceed that latency budget to meet the overall service level objective of that. Yeah. Yep. And that's something that, I mean, whether it's something that a lot of folks will run into, or just if you understand the business use case for this particular application, then you understand that the, the latency budget becomes a determining factor in the architecture.

Um, either way that's, uh, yeah, absolutely. And because of that, we made certain choices in the architecture too, where we decided not to pass these big message bodies. And instead we just passed IDs to the actual, uh, code or the X, the XML payload. And so we, we, we, we dramatically reduce the cost of syncs, uh, of, uh, serializing and deserializing this large payload. So we optimized as a result of that choice. We optimized the architecture to an engineer to achieve, to achieve that, uh, objective. Yeah. Okay. I think that that's, that's helpful. And then also starts to get us to our next question, which is around, you know, if you were to do this differently, do it all again, you know, what would you do differently? And, and what, what do you think you would do the same? I mean, I, I think I would pretty much do, it would be just repeat. I mean, it was good, but the only, only difference is, uh, we have that experience with us. We have that, uh, you know, what we did, so that will help us. So we don't really have to, you know, be in that position that Rohit was describing, right, where it gets uncomfortable. So at least we know some of those pains we can, uh, you know, upfront address, but some things that I would want to call out that we would continue to do, uh, you know, is, is we would definitely want to do as driven development. We definitely want to do their programming, uh, which we have, you know, a kind of practice, uh, and, you know, not just, uh, survive, but tried as well. Um, and, and of course, you know, we discussed, we talked about it in the earlier segment where, um, the granularity or, so just keep it simple. Like, so we said, okay, let's do it. Uh, what we used to call it as a shoddy viable product SVP, just keep it, you know, get something, uh, to be there, start, start building that, and then, you know, improve iteratively, um, and then build a lot of automation.

So those things I would definitely would, uh, do it no matter what project, uh, uh, it is just, not just legacy transformation, but any project. So if it means I have to redo this project, I would, you know, pretty much do it, uh, again. And there is, there is things like, okay, uh, one thing is, as Rohit said, the thing that I would do differently, uh, yes, we did testing to make sure the rates match, uh, but there were things, uh, you know, like, how does this fit overall into the ecosystem? So as we kind of started integrating this with different applications, different things, uh, we had to kind of, you know, figure things out or they were like, Oh, gotcha moment. Like, okay. Uh, we, we had to go back, you know, tweak things, come back. Um, if we were to do it again, I would probably, you know, kind of have some focus on making sure that we do, uh, overall, I mean, we have a lot of, uh, you know, unit testing and lot of integration testing within this immediate, uh, you know, application vicinity, but make sure that the application or the product that you are kind of, you know, fitting in, uh, especially in legacy transformation, we do a lot of end to end regression testing as, uh, and often as you can, and as early as you can. So it may not be your system that has an issue or that that would create problems, but how those things are consumed or, you know, the moments and sometimes they are like, so innocuous that you don't even realize that, Oh, this is what, you know, how the others are consuming it, or they are kind of reading it. That is the only thing that I would be a little more, uh, you know, put more emphasis on, but there is not much that I would change from what we had done. Okay. I would like to someone like Rohit being there. And if I could repeat, like, I would want, you know, someone like him to be there next time we start as well. But that's, that's one thing to see our team also underwent some churn, right? We actually remember like, as the project went on, we lost a few key members. If I had to redo it again, then we, we, we wouldn't

allow them to go. We had an awesome team to begin to start with too. Um, as an, but testament to the architecture, even when we lost some talented people, like new people came in and it's still, it's still grew stronger from there. But, but when some of the critical folks leave, it does leave that vacuum for a little bit. So I guess that's just life. But, um, there is kind of just a statement on, on, on, when a project runs, runs long like that, your team composition will also change. As long as there is, there is an anchor, strong anchor, uh, like Viraj, then things work out. Uh, yeah, but we also had like the, you know, the, the biggest thing that, you know, we had was the pair programming because even though people leave, they don't take knowledge with them because the knowledge is spread across, right? So everyone knows about everything that they may not really be like super experts in it. Like, oh, I know, but everyone knew, everyone knew, uh, enough about it to kind of go into it, fix it, or, uh, look into it and understand it. Uh, so that was the biggest thing. Everyone understood different components. Everyone understood the decisions. Uh, it, it is a team, you know, effort where we were not saying, okay, you know what, we are making a decision. Go ahead. Just developers just implemented. We kind of collectively, you know, discuss decisions. Uh, it was a collective responsibility. So as to say, hey, and as a result, a lot of, uh, you know, team members, when the experienced folks or those who have been, uh, you know, around left, new came in, it was not difficult because others were there to onboard them. Uh, and it, it just felt like the, the anchors, you know, help, um, in, in that sense. So, yeah, it, it sounds like from the things that you listed, you would, you would, you would keep were some of these factors that really help build resiliency against things like departures from the team. And, and I think that that's something worth calling out because like no one's plan for building

an app or modernizing an app, no one's plan includes X amount of churn on the team doing it. Like that's, that's sort of this, it's always a surprise. And yet if you were to go and measure how all of these projects work out, you would probably find a statistical measure of you should anticipate, depending on this amount of time, you should anticipate this amount of churn. And, you know, it's not necessarily good or bad. It's just that this is statistically what you should anticipate. And like, as the time goes on, then, you know, it's not necessarily linear, right? You can probably expect a slightly increasing rate of churn over duration, which just has to do with life, as you said. And then if, you know, if you're statistically, you know, if you're out of bounds of that sort of standard deviation, then you, you might have a problem. But no, the point is no one plans for that. And yet it's a reality. And so having things like, you know, pair programming and, you know, test driven development helps sort of bake in resiliency against it. And then just to make sure I capture everything else, sort of keeping it simple with sort of the shoddiest viable products, MVP approach, building a lot of automation, and then adding in the end-to-end integration testing early and often would be kind of like the, these are the things you don't want to skimp on. So, and this, I've got one more question that I want to sort of shift gears to, which is now starting to look at the stack, the technology side, and understanding, you know, how you made decisions at selecting different parts of the technology stack from, you know, .NET and, you know, were you using the steel toe components? And you mentioned you had a cloud platform. I assume it's Pivotal platform, you know, how did that get selected? You know, what infrastructure is it running on

underneath? Is it in-house? Is it on a cloud provider? And how, how did those decisions kind of get made? Well, as far as .NET is concerned, within at least personal lines, we have .NET as a standard platform. Though we have Java, we work on, you know, different things, but looking at the skill set, looking at, you know, the team strength, .NET is our go-to platform. If it is new thing, unless it's something else fits in better, like if it is something that we should develop it based on the availability of either tools or whatever, you know, kind of blends it into node or something else, we tend to, you know, kind of, so .NET is given. And we, in this case, we decided to go .NET Core because we wanted to, you know, kind of deploy it on the cloud. And, you know, tomorrow, if it is a multi-cloud platform, we can't always say, oh, we have to be on a Windows stack. So .NET Core was given. Steel 2 was, again, something, you know, as a best practice that we learned from, you know, working with Pivotal. It's a great, you know, tool, something that will help you build your microservices. I mean, you don't want to reinvent the wheel, you know, just to describe Steel 2. Like, there are things that it provides that otherwise I would have had to write, like, you know, all this connection, you know, health check, what have you, right? So why would I write that if it is already, you know, provided? It makes my life easy. And if it is, you know, something that is constantly enhanced and updated. So that was also one thing that we definitely wanted to, we started with this. And we wanted to now kind of continue that all the new things that come up, we say Steel 2 is must, you should use it. Other things, you know, in terms of the Pivotal Cloud, it was a thing that we had in-house. I don't know when it was chosen or how that decision got made,

but it was not specific like, oh, we want to do this and this has to be on Pivotal. But it was a, you know, on-prem cloud platform that we have. It had been, you know, kind of, we had been working on slowly to kind of move application, harden it and kind of make sure that we can leverage it the way it was, you know, intended to be. So a lot of things that we did were like, kind of no brainer, like, okay, yeah, we have to, there has to be some of those things that we had to use. A, because they were the existing tool set, or because, you know, they were the right tools for the things that we were, you know, trying to do. Okay. I don't know if you want to add anything, because you have been involved in, with Pivotal and with Travelers for a long time, so. Well, and if there's any other kind of technology selections that were made, you know, that I didn't mention just because I don't know about them, you know, what else kind of went into it that you can share? So we used like, we used .NET Core for most of our development. We used Steeltoe as one of our frameworks for connecting to services on the platform for, for health checks, for service discovery, for actuators. Then we used the tools that the Pivotal platform at Travelers has for logging and metrics and monitoring. So yeah, we also used like Rabbit and Redis. Like, these are like, you know, marketplace services that we have within Pivotal at Travelers. So. Rabbit was our, Rabbit was our messaging broker, right, where we exchanged all our messages for one of the rate codes. All our microservices communicated through, on the RabbitMQ messaging, on the queues to communicate across one another. And then Redis was the cache to hold the big payloads. And so that we could only send references across. We didn't have to

transfer the whole payload. It was Rabbit, Redis, Steeltoe, and .NET Core, kind of stack. And so I imagine, oh, go ahead. No, I'm sorry. I mean, that's pretty much the large, you know, part of what the toolset we had. Then we had something like, for some of these, you know, like when we had some abstract rules that we had to build and validate, we used, you know, it's an old framework called fitness. But we just wanted to make sure that we have automation all around. So we just wanted to make sure that we do use that to validate some of these rules. So anyone like, you know, a business analyst could say, hey, I want to try, you know, this variable, this variable, and see that rule is applied or it is able to, you know, produce the same outcome that I'm intending it to. And we had that as a part of our build pipeline where, you know, we would want to make sure that it passes the fitness, you know, test. And it's a wiki-waste, you know, tool. It's old. I mean, I don't think so a lot of people use it or know it, but it is something that we had. We used it as well. And what were you using for the build pipeline? We were using Jenkins for our CI. And for CD, we have a tool called UCD, urban code deploy. So Jenkins was hooked on to UCD and Git was our repository. So we used code in our check-in in Git. And then whenever we checked in the CI/CD pipeline, we had like other things that were looking at code quality. Like we had SonarQ, which was looking at what it is. And then we have BlackDuck, which looks at vulnerabilities and stuff like that. So those were built in. Those are standard. There was nothing specific to our project, but they are like a part of our overall pipeline.

Yeah. And I mean, just even kind of what you mentioned about, was it called fitness? Yes. F-I-T-N-E-S-S. Yeah. Right. Like the Nike fitness training app or something. Yeah. I think if you search on fitness, F-I-T-N-E-S-S-E or something, it's a wiki-based, you know, thing. So it's very simple to configure. You just need certain setup. Like one of my team member had that experience. He suggested, hey, let's use this for, you know, this rule testing, like rather than building something or rather than, you know, kind of having someone go through the code or run unit tests just to validate, you know, like if he or she is able to test the rules independently. So we use that, like, you know, just because we had that tool, you know, or somebody knew about it. Yeah. I mean, I think, and also like the idea of being able to put that in as part of your built pipeline so that any of these, a lot of folks think, okay, well, yeah, we've got to run our unit tests and our integration tests and we have that automated. And it's like, well, if you think about the full path to production, there's these other points of validation and the, to the extent that you can automate those, you're going to make subsequent code changes that much more frictionless. So I think it's always interesting to hear like, okay, what are, what are some of the different steps and checkpoints people do have to make sure that they've got production ready code and that they can make those changes easily. So, okay. Thanks for the link. I'll, I'll include that in the show notes. And of course I have to ask a little bit about rabbit. Cause I spent a lot of time with that team, you know, you, you, you mentioned the, um, kind of the latency sensitivity, uh, and how making sure that, uh, the different microservices, they're all communicating via rabbit. Were you tuning that, uh, the, the, that sort of

set up with rabbit so that you were getting kind of the, the, the lowest latency and highest performance results or what were you doing on the, on the, on that side, if, if you're willing to dive into that? Um, I don't think we did a whole lot of tuning. There are certain things that we did is, uh, you know, like the application that we have is important, is critical. Uh, so we, we had to make sure that what we use, uh, is on demand or, you know, kind of dedicated in, you know, in an, uh, instance of rabbit, it's not a shared instance, uh, so that we don't have a noisy neighbor, you know, who kind of just influences, uh, you know, or kind of impacts. Um, we did use like the three node cluster so that there is some amount of resiliency, uh, you know, into that. So if one server goes down, one VM goes down, and we, we are still able to carry on and, and as you know, kind of Rohit alluded, uh, we were not really putting in a big message, uh, or like the, the message size was big, but we were not putting that into rabbit. So all we were doing is just a notification with the ID saying, Hey, you know, this is the ID, you know, go grab it from the cash and work on it. So that way we kind of kept it, uh, minimal. Uh, and, and then we also made sure that our pre-fetch count is set enough so that you don't, you know, constantly hit, you know, rabbit to get work. So you would say, okay, my pre-fetch count is 10 and, you know, take work on 10 things and then, you know, go back and get some more. Um, that pre-fetch count is something that we had to try out, you know, a few times, or, you know, kind of make sure that it's, it's appropriate, uh, and then we are not, uh, hitting. So these are some of those things that, that I am aware of that we, you know, kind of, uh, had to learn, had to work on. But, um, other than that, I can't, I'm still thinking, but I don't think we did anything, uh,

differently. I mean, Rohit, do you know of anything that you? Yeah. So, uh, Viraj, remember we used the dead letter queue handling from Rabbit? Oh, dead letter, yeah. We had the DLQ handling, which we used. Then we also measured end-to-end latency with distributed tracing with Zipkin, right? So that we could tell for each where, what portion of the entire latency was being spent in each, uh, microservice. What was that span worth? What did we do distributed tracing? Or am I not? I think it, it was not, Zipkin was not, uh, available for .NET Core. So we, you know, we were hoping that it would be, but it was not. So we never got to that point. Um, but, but we use DLQ. I remember that, right? Yeah. Yeah. Dead letter queue for sure. Uh, the other thing that we, you know, since, uh, you talked about dead letter queue, one of the things that we had seen in some cases, uh, uh, is if the message stays long in the queue and we could not really, you know, always reproduce that or say, Hey, this is why it happens. Um, we would lose or, you know, sometimes the transaction would just time out after like 20 seconds or 30 seconds because it was just in the queue and every, you know, then the calling system would time out. So what we did was we implemented time to live, you know, kind of on the, on the messages. So we didn't want to really, you know, clutter in, even in, in, in accidentally that, Oh, messages leaving in there or, and that they're not getting cleaned up. So we just make sure that, um, beyond a certain time, the messages should not be in, they just get moved out to dead letter queue and they handle, they are handled as an error. So we also had great support from the business team, right? Remember they, they looked at all the validations and made sure that the, like all the business rules for mapping of all those policy things were correct. So I want to give a shout out to that as well, because like there was a lot of business rules in terms of just like mapping and derivations that were on the ZOS side that we

ported to distribute it. And without their help, I mean, that would not have been do a practical, uh, possible. Right. Yeah. Sorry. That's another team inside, uh, travelers that was just good to collaborate with. Yeah. These, these, this was the, you can think of these domain as like the business analysts, the, the, the ZOS domain experts of that system that knew and had that had in some ways worked with the system for a very long time. Yeah, no, I think you're right. We had at least two individuals, uh, who were part of our team, um, who provided that it was one who was mainframe, you know, expert, not just mainframe, but, uh, she had worked on a lot of, you know, legacy apps, um, including, you know, the one that we were, you know, kind of moving from legacy. So she was working with us all throughout just to kind of help us, you know, kind of answer our questions or in some case, you know, guide us saying, Hey, and in, as we said, you know, validating certain things that we were doing, making sure that, okay, yes, they indeed are the right way to do. So there was that, uh, you know, tremendous collaboration and not over the email. That was, you know, the thing, like, it's just not like we were sending emails and, you know, we got, we got response. We were sitting right next to each other. Uh, we were sitting in the same, you know, uh, area where we could just walk over or we could, you know, uh, request for a quick huddle and say, Hey, let's talk about this. We discovered this. How do we fix it? So we could, you know, address things quickly. Interesting. Yeah. I think that that's also something that, you know, sometimes folks thinking about co-locating, co-locating teams and whatnot, that that's a good piece of feedback for folks. Um, you know, the value of that. And one more thing that I just want to quickly add was, um, since we were moving at a fast pace and, you know, we were kind of doing things,

uh, we also had a focus on automating things, right? So as we described earlier, uh, that we used to compare rates on, you know, the old and the new system, again, you know, we are talking about not one, two, or, you know, 10, we were talking about like thousands of such transactions running, and there is no way you could do it without having some automation. So we wrote a tool, uh, you know, which would kind of automate. So we just had to say, Hey, here are the, you know, here is a database, go get that, uh, you know, payload, uh, fire these, uh, you know, uh, requests, compare the output and, you know, uh, spit out the result as to what is the accuracy. Like it's 99% matching a 5% and use that feedback. Like, okay, if it didn't match, what were those transactions and why did they not match? Uh, and maybe, you know, in, in many cases we discovered things that, Oh, which we missed or which we didn't do. Uh, so it helped us, you know, kind of fine tune and not fine tune, but in some many cases, fix things that we would have missed and, you know, went in and we could have gone in production. Uh, where I'm coming is automation is critical, you know, especially when you are building something, um, like this is an, in legacy, uh, you know, thing where you need to compare, uh, you can compare at scale. And if you have automation, it just helps you, you know, kind of deliver things, um, if not faster on time that you promised. Yeah. Um, yeah, no, that's a great point. And that's, you know, in this case, it was very specific to the application that you were working with. Uh, but it was worthwhile to build some automation specific to that just so that you could make these changes with confidence. So in some ways it's, it's, you know, to me, it's not unlike how you need to write some of those tests first to know that you haven't broken the functionality. You need to automate those things so that you can know that you're doing it efficiently. Um, okay. Well, I'm just, uh, trying to be mindful of time. This has been really, really

interesting and useful to kind of understand the, the process and the tooling and, uh, the challenges along the way, the trials and tribulations of modernizing an application off of the mainframe into a .NET core stack, um, that's, you know, ready to scale on demand and making use of a lot of modern technologies. So, um, do you have any last words on, you know, how this is delivering value back to the business so far? Um, yeah, I mean, they're happy. I mean, at least from what I see that they are making plans around kind of saying, how do we use this, you know, uh, not just in like today we are doing it for one line of business, but they are already saying, Hey, how can we do this for the next line of business? Uh, you know, we want something similar. Um, and we are starting with some other initiatives where we are going to do, uh, what we have done in this case, uh, build a brand new, you know, thing in this case, in that case, it is not going to be a legacy transformation. So as to say, um, but it is modeled on, on the same thing. So again, in a day, if they were not confident that this is working or if this is not something that is producing value, um, no one would have really, you know, kind of, uh, supported that, accepted that. Um, and, and then I can, I can see that they are making plans for what we have built to be used, uh, more effectively and in, you know, in a much wider, you know, thing we started with something small only for a few states and only for, you know, certain things. Um, now, you know, we are kind of expanding it gradually, you know, making sure that we soak in all those things that are discovered, uh, and then we are going. So I think I do see a lot of excitement and a lot of, uh, you know, um, uh, opportunities that business, uh, can leverage using what we have built. So, yeah, I mean, that sounds like there's a lot of confidence in the

process and the results. So, you know, success breeds success. Um, Rohit, any, any last words from you on folks who are kind of looking at making these types of changes and what you would recommend for how they get started on those engagements? Yeah. Get started. I mean, start, that's the big thing. Like, uh, don't expect it to, to be easy, but if the organizational commitment is there, uh, you will make progress and over time, the, the results will be manifold. So also understand that it's a journey, what the discussion about the results and outcomes that we are having here is a result of a two year project, right? That we started. Um, this is something like, it's like, we're discussing, we're reviewing a movie here, which is just two hours long, but it takes like eight, eight, nine months to make, right? So yeah, uh, it's a, it's a process. It's well worth it. Um, and you realize real costs. You realized all those three things that we discussed at the beginning of the call, right? Uh, you realize cost savings, you become more agile. Uh, you now have a set of technologies that your more modern workforce can work with. So, uh, if cost, uh, skills and agility is what you need, then you need to put forth a program like this to get more details. Sure. Sure. Listen in on our June, not June, January 23rd, uh, webinar. So we'll get into the deep details and cover all the gory, uh, bits of the promotion. And with visuals, those are always fun, you know, nothing like a good, uh, you know, flow chart or network diagram or something so that we can wrap our heads around what the heck is going on with, uh, these microservices everywhere. All right, gentlemen, thank you so much for, for joining us and, and sharing your experience. This has been, uh, tremendously insightful. So thanks. And we'll look forward to the,

uh, the, the webinar in January 23rd. Thank you. Thank you.

Tanzu Talk · 🍕 Get more pizza.