How to change mainframe apps into microservices, modernizing mainframe applications with Fouad Hamdi

Episode 261 · Fri, 28 Apr 2023 14:10:34 +0200 · 36:59 · SRT · archive.org · all Tanzu Talk

well people are always asking me about uh mainframe modernization and and well let me think about this always is that true yeah it very frequently comes up i think especially like i was i was realizing uh in this past year i don't know if you've experienced this but in this past year people i talked with for years have always been talking about like moving to the cloud and modernizing their stuff they really wanted to do it but i've noticed in the past year people actually have numbers around it like like goals and kpis they're supposed to reach which seems like it's it's uh it's happening more and more so you wrote a uh you wrote a blog post this is actually last year from may of 2022 when i was uh when i was looking over i don't know well i don't even remember what i was reading about legacy applications and i came across this on our blog and you had a pretty uh what i think is a a great in the sense that it's brief and just deep enough to make sense like overview of of uh i'm sure along with many people like you worked on um helping modernize uh uh you know a mainframe application uh and it was a good overview of a lot of the practices i've heard we do in swift and things like that with some uh with some nice diagrams so i thought it'd be fun to uh to talk to you about that why don't you introduce yourself first so i am i am fwad and i am working uh in tonsu r&d uh today and uh since a few months ago and before that i was working in tonsu labs uh in france in paris and i was involved in many engagements where we used to modernize legacy applications and mainframes are uh some of the kinds of application that that we we are used to modernize there do you see a lot of this out there like mainframe applications that people want to and then actually are modernizing or you know does it happen rarely so personally it started like uh i think i was involved in that kind of engagement like uh in 2019 like already three

or four years ago and and at that time there was not really many mainframe uh engagements going on but since a couple of years it's i've seen in labs uh uh that there are more and more customers asking to to to dig into that and to help them modernizing the mainframe so give a give us an overview of like um both both the example as you say engagement here but like i think it's pretty representative of what what you see let's call it the business motivations for modernizing a mainframe you actually cover it yeah pretty well right from the beginning like it's a 30 year old system with 12 000 programs that's relied on by 3 million users you get into like what the business motivations for doing it are and i think i think going over this is always very important because like you know for as much as mainframe people used to call us distributed people might sort of poke fun at mainframes and have them be old like the thing about a mainframe that's been running 30 years is it works right and it's it's worked very well for 30 years to survive through all of that when companies have mainframes they usually work fine uh but but for for this example one of the main issues that the customer had was the that they the lack of knowledge to maintain and to make the mainframe so that was one of the key key cornerstone basically that they they they wanted to get rid of to be able to have people who who developers new new teams that uh who could handle the business that is implemented in the mainframe but using modern modern technologies uh so that was that was one one of the aspects and and uh the second big one was uh to reduce infrastructure costs basically because having a mainframe processors running cost them millions of euros uh and and being able just to move some transactions that were running uh into the mainframe

into new application would make them save a lot of money so they wanted to start to do that the goal was not to get rid of the whole mainframe like uh uh as fast as possible it was more like to start extract few a few things step by step and and being able to do like to have quick wins and a quick money savings by just doing that i wrote this little booklet last year with with our colleague mark zottner about uh you know about the swift method and and like more broadly like uh we called it the legacy trap that was our clever name for but more broadly like from a for lack of a better phrase a business perspective like what the deal with modernization is in doing it like we went over like what makes software legacy software and i mean you you i think you hit on one of the to kind of summarize the two of them like with mainframes especially there's the like you it's you've forgotten about it like you lack knowledge about how to do it and you know i think i think the how that manifests itself and like something that the organization cares about is that you can no longer really change or add to the functionality of it or and and sometimes it's also performance right like if you've forgotten how a system works sometimes when you have different performance needs like you have to serve a whole bunch of different people or you have to do it in a different medium like you have to go from a desktop app to like a mobile app or something like it's kind of no longer fit but i think i would imagine the one that people encounter a lot more is like you just have different features that you want to add and it's kind of not possible right and because because no one really remembers not only like in this case like how to program in cobalt and also not only just like how the programming works but like one that's especially like terrible is like no one remembers how to test it yes to make sure it doesn't break things which is terrible yeah that's true so so uh they also have to rely on external companies to to maintain

this this knowledge and and that's a big a big danger for for such companies like they move the the business loader they externalize the externalize that business knowledge that knowledge to external companies they don't have much more control about on it rather than uh uh in the past they were able to control the their business inside the company so so that's why that's one issue for for this company and and uh i saw another thing that uh then another issue that they have is the development life cycle like you mentioned uh the capability to to deploy new features or to fix existing features uh also takes a lot of time uh just to deploy new things uh it's a they have heavy process they had heavy processes they had to set up downtime periods like to to stop the user the users using the application to be able to deploy your simple feature that was very painful for them their tech debt finally caught up with them so you mentioned one thing i'm interested in in hearing more about which is um and this is part of like the the the method that you know tonsu labs and pivotal labs people have developed not only in legacy modernization but just in apps and platform engineering like in general which is like starting small basically is is the way i think of it right instead of oh we're going to modernize and move everything on the mainframe like more so it sounds how you all started was like let's start with one thing let's start by just to use a word that's common like little slices and slicing away at it was like the the customer comfortable with that often there's just like well why wouldn't we do everything at once but i think there's also often like an overly ambitious business case they have to deliver on like if we're going to take on all this risk and like spend some money on this we need to have a big payoff at the end of the year whereas when we go again with this start small and slice thing we're kind of like saying yeah there's not going to be a big payoff for like three to 12 months did you encounter

in this case where you kind of had to sell people on starting small and slices what what was that like yeah that that's usually uh hard to convince people uh companies about that customers about this but but in that case for example they already tried it in the past so they already tried to rewrite and they they and they felt the failure the the pain to do so and and that they they were not happy with the result so that's why they wanted like uh to experiment our our approach and actually before before working on the mainframe we had past engagements with them where they trusted us yeah basically we built a trust relationship with them and and that's why they involve us in try helping them modernizing the mainframe it was a kind of some kind of challenge right like uh okay uh prove us that you can use your practices to to start modernize you something in the in the mainframe and see and see and check if it's better than what we did in the past or is it worse or oh yeah and then to can to kind of beat some comparison charts between uh uh our approach i guess this is what like you know labs thinking would say is like well let's let's do a small project or a slice to see if we can also do this with mainframe stuff right let's prove that that we can do this out uh in kind of like uh it's not risk-free but a lower risk way when you're looking at the application like how do you start with that like how do you how do you start like comprehending what this application is kind of like figuring out what little slice to do kind of kind of walk us through that process yeah sure um so basically we we we use one of the tools in our swift uh swift uh swift box swift methodology box that is called event storming basically and and what we do uh is a we run a workshop with uh uh every everybody uh all the people that are who are involved in the mainframe who have knowledge on the mainframes and and that can uh imply having people that are

at a high level of the hierarchy of the company who often uh were developers in the company and know and know about the mainframes but now they've changed their position there are no more developers and they and are actively promoted and they and they have but they have knowledge of this so we have to involve those people in in this workshop uh until the develop and also the developers and the testers we have to gather everybody uh in the same room and and basically build a visual representation of all the of the events that are flowing through the system like basically like uh uh everyone brings bringing uh his or her piece of knowledge uh regarding all the things work in the system in the mainframe and we have to to merge all of this uh uh so it's a sticky notes exercise basically we have gathered in a piece in the room and we put stickies and people put stickies and then we have to merge all of this to make sense of all the the flows that are uh that we have in the system and and from them and from there we we can we start to to draw boundaries like uh identify subdomains like uh the comp the like uh okay this makes sense to gather those events together and to call that uh i don't know like the the accountability subdomain and together and and and build that kind of uh um identify that kind of subdomains and and from there we can also we also uh we can also have a an exercise where we with the uh where the people in the room can uh set the value uh of each of these demands right right sort of like the business value like how much does this matter exactly how important is it exactly and also try to identify the pain points where where they have the most the pain points maybe we we would like to start there or not depending on the context because uh depending on where you are in your mainframe modernization

modernization journey maybe you want to start with something that is small not too hard to implement to validate the the process that you have in place but it has some some business rules so so the selection itself depends i would say depends on the context but the tools uh that we are using as are basically the same put some values on the thing uh identify the complexity identify the the business value and try to and prioritize based on the context where you are working what what are you looking what what is your goal with your modernization and then right select the appropriate uh let's say context or subdomain context and then and then from there identify uh user journeys and and the slices that we could start with and so so that gives you a list of just as you're saying like here are technically feasible and important enough like components we could work on modernizing because again that as you're saying the goal is to let's let's find something we can modernize that takes i mean i think the standard is like 12 weeks or whatever but that takes a small amount of time yes uh so that we can as you know just to summarize again that that we can figure out if we can do this right and at the same time actually do some of it and and also like learn how it's done i'm interested to like test out some of my theories uh you know with you about that is i feel like that exercise that you went over the kind of like you know the event storming doing like the the finding the domains all of that workshopping stuff like well i mean there's not not an assumption but the kind of the unstated thing in doing that exercise is that it doesn't that that that system view doesn't exist right like that like like while while each of those individuals might like have their own view of what their part of the system does that very often especially with like in this case like a 30 year old application um no one really knows the whole let alone everyone

knowing the whole and like what the handoffs are between them um so i imagine that's a huge part of it it's just like as as mark will call it it's like architectural uh archaeology yes like discovering the system exactly yeah that's a huge outcome of the of this workshop indeed like uh having this view that probably nobody's at the beginning of the workshop had in mind and and align also all the stakeholders on the on this knowledge so of course the what we get at the end is not exhaustive like we sure you know we don't go deep very deep in the details of every demand but at least big picture everyone has a view of the big picture and that is uh that is very i'd say people when they they after this this workshop they felt like wow i understand what i'm working on now like yeah well i didn't imagine that there was so much complexity in this part oh like uh there are many uh that cannot we had a a lot of feedback like this like after the it must it must feel nice on both sides to actually like i mean you know in our trade learning is always fun and so like like like to have helped you know the the group as a whole to like have learned some new things especially about something you're so close to like your own systems right and you know the the other thing that like i think and and you know in general this this methodology i've i i shouldn't say seen because i just record things and make slides but like you know i've i've i've heard secondhand from labs people that whether it's new applications or existing ones right like the other i think the other thing that's valuable about it and and to your point of not to your point but like the positive side about of it not being too detailed is that that's by design because the purpose is also to capture like again to use the phrase like the the business flow of it like the purpose of this application right which is often like not fully reflected in all the technical systems like you know and and i think by by doing that event storming where there's like one event that you trace through the entire system like

i don't know just to pick something like uh you know authorizing that something happens whether you're authorizing a prescription or authorizing like some forms being filled out like that whole authorization thing is an event that goes through the whole system and like i always joke that like everyone most people know the boxes but it's those arrows that people lose out on like like the connection between the two of them is like where a lot of lost uh knowledge uh typically is so you get the slices in front of you and so like what uh like like what slice do y'all pick like how what do you start with as you document in the uh in the blog post in the in this example we started by uh uh using like uh by working on a a slice uh a simple slice basically to to to be able to build something very very fast and see if we could deploy it in production uh very quick and evaluate um and identify all the trouble that we could have uh as fast as as quick as possible instead of waiting to build something more more more much more ready for the for the users the goal uh so so basically we we use the read read transaction written actually even before that sorry even before that what we did is we wanted to to start uh uh experimenting on what we call strangling the the mainframe legacy system to be to be sure that um so the strangling uh an application is one pattern uh where we try to basically to capture the calls like the requests that are done to the legacy system we try to capture them in our new the new system that we want to build to see if we can then uh if we could then uh rewrite them like uh modernize them in our right system and so basically it's kind of uh adding a proxy a gateway between yeah legacy system and our system uh so we we had to start with that because because there were a lot of uncertainty

about the architecture uh at our customers because our uh so if you if you look into the article there was this schema where we have a like a front end a system and then the back end system and then the bus that that redirects the calls from the back end to the mainframe and we want i don't know if you remember that part and we wanted to to basically to integrate a proxy gateway uh between the bus system and the mainframe basically like uh uh we wanted to see if we could capture the calls from the bus to our gateway and then redirect the the calls coming to our gateway to the bus again that would call the mainframe just basically to see if we could capture things there and yeah i mean it's essentially a uh like a yeah i guess you could call it a person in the middle right like exactly you just add a layer there and then at that point well i guess one like you experiment with like we can actually connect to a mainframe and just be a pipe right exactly and and just getting to that point with authorization and figuring out how to do that like and making sure you have the right character and coding and all that stuff is probably plenty of work basically to do quote unquote nothing right exactly and it is very important also to it was very important to do it first because uh because we realized that it failed basically we were not able we were able to build the gateway and to to think but but the thing is that the bus the bus uh configuration was was maintained by another team uh all right yeah we didn't have the the bandwidth to work with us basically to to configure things to road direct to our gateway because um if they had to do that we would have to write like specification and it would take six months to implement this just to reconfigure the bus uh to so so that was a uh a very important experimentation to have to to validate that to be sure that we could not do that so uh and it forced us to pivot and to and to find another way to to stronger the mainframe and basically we uh i think i explained it in the

article basically we there there was a ldap in this architecture that that is used to to to uh by the backend to to know where uh to know the routes that can to call on the on the bus when they want to call a mainframe transaction so we changed these routes to rather yeah let's get it to to a new uh to our system let's say to simply yeah it was it was fortunate that that but the that bus was in place right like that that that was kind of already a uh uh uh sort of already as you're saying a proxy or a gateway or whatever that you could have been so we were lucky that's why the context where you are modernizing the application is very important because yeah something that works in one context might not work in another one and uh and uh yeah we don't try to come with uh at labs at least we don't try to come with already keyed solutions like okay we're going to to do in your context and study the problems that we're going to meet to face by doing the things uh rather than just diagramming things so you so you start off so basically you're fortunate that there's a like call it an enterprise bus but a bus in place that's already like and and was that like was that like web sphere connected to the mainframe or like what what was the the bus i forget i think the bus was oracle esb and yeah the front-end and backend stuff were deployed on weblogic if i remember yeah yeah that would make sense good old j2ee stuff back when there was a two yeah and and so so essentially you put that uh you put that gateway in there we'll call it that is again putting it kind of sarcastically does nothing like like all it is is to make sure that you can have this pass through and then i imagine you know the goal of that is that as you have new applications or applications that that want to talk with the mainframe or existing ones you just shift them over to using that gateway right exactly and which should more or less be the same so what what's like the next step after that you get the gateway in place and then and then how do you uh how

do you do further slicing so the next step was the next step sorry was to um to to work on the one of the uh one of the business transactions that was um responsible of reading information from from the uh mainframe databases from db2 and dl1 databases and so basically we start to we started to write uh a new a new microservice let's say let's call it like this because it's a fancy word that contained uh an endpoint to to read basically uh information from the legacy database and and so uh if i try to describe the flow so the front end will call the the back end that will code our getaway and our getaway will call our new uh yeah new endpoints and return the result to the front end instead of calling the back the the mainframe to our to our uh application new application the other thing that i think is worth pointing out or i mean you said it but calling out even more is that it was just a read-only operation so it's like again minimal risk right like like like most risk comes from writing or changing the data versus just like yeah seeing what it is is not that big of a deal yeah yeah and uh it also helped to build confidence within the team like uh right because we didn't have much knowledge on mainframe stuff and that's what to learn to have learnings because for example uh uh in the they were using db2 as a database but they are they were also using another database that is called dl1 which is quite old and it that uh uh that is a hierarchical database uh not a relational database and that uh there was no no tool to access the database for modern applications like to be able to read uh okay directly in this kind of database actually it's not really true there was one tool uh uh not open

sourced like and and there was a uh a team who were studying the ability of this tool to to be used in modern application to read in this legacy database so so uh that allowed us to learn things like that even even a a simple a simple slice like this was very beneficial for for the team so then you know uh eventually you want to actually like change the data and write to it so what what how do you all get to that so uh so this is yeah this was the moment where we introduced this is the moment where we introduced a new database for our new application to give it the responsibility to be the owner of its own data let's say and and basically it started the same way than the read slices like we we we configured the ldap to route uh to the new to a new one endpoint that is responsible to write uh yeah that in the database and and what we did first is to um so is to to set up this new database and to and to basically to trigger to uh to to to parallelize the the function the fit let's say the functionality i mean by that like we we are saving the data in the new system but we are also saving the data in the old system we also calling the the all the transaction that we are rewriting in the old system so it's about it's a parallel right like uh we are writing in uh both in the old and the new system uh why we did that is because uh uh we wanted to um because there are a lot of data that are that have to be written let's say and uh and a lot of business rules uh in the old system that are used to to write data and we wanted to implement those business rules incrementally because they are uh this is the way we usually work we don't want to to do a big big bang rewriting uh and and so uh basically the the source of truth remains the old database while we build uh while we incrementally build uh implement the new the

business rules to write that time right going back to the beginning you've got 30 years of business rules yes and and like and like you know who knows what they are really yes yes there we find surprises we have surprises when we dig into those things that are not used think that nobody knows what they are for and that can happen yeah yeah and and then essentially with that that uh like that comparator pattern you have i mean well you are like testing in production which is always a scary phrase to use with people but like you're you're making you know as long as you're really good at accounting for this as you implement those new business rules um like in your new database you actually can track if your assumptions about them are correct by like what the the comparison looks like and and then also i would imagine you also i mean it's annoying to have two copies of the same data like eventually i imagine you have to consolidate them but then also ongoing uh other systems that are using that data that have not been modernized they can go on functioning right like they they can use that old data which which again it fixed into this notion of like let's do the smallest amount possible without without without without uh having to mess with the rest of the system you are right you're right um totally um so yeah this comparator that we built uh to to validate that what we are writing in the new system is the same as the same meaning that what we are right than what we are writing the old system uh also helps build a lot of confidence uh um in the team but that's when the whole company because is a very sensitive subject to touch to modernize mainframes there uh because we are touching 30 years of yeah like you said business rules and yeah things might break and and and having this comparator mechanism even if it's complexify a bit the architecture uh helps to build that confidence in the company that we are not uh doing uh we are not introducing uh too many i want to say i don't want to

say no bug because they are they can bugs can happen anytime but it's just the risk of things the risk exactly yeah and and like you said after this step when this confidence is built the next step would be like to to check to swap the the the source of truth for the data yeah to to give the the source of truth to the new application and then to to synchronize the whole system database with the new one to make the other application that use the same data still more and that would be the next step after building that the confidence path how long do you think that takes like do you think that's like like three months or is it like two years that that you kind of just like let let those two databases kind of like simmer and merge together yeah i think it take months uh and years just for this example it took two years for them to rewrite the transactions like and to have this coffee and to be able to to move things to change the source of truth so it depends on yeah the complexity of the transaction and uh but uh yeah it can i will not evaluate that in weeks that's really interesting to me because i haven't thought about i haven't thought about like this kind of scenario but in general the other thing like when i talk with large organizations about modernizing or doing anything is one of the other first things in addition is starting small that that i try to emphasize is like however long you think this is going to take you should probably double it and and it's it's not just the usual like programmers can't estimate and it takes a long time but it's it's more of i think a lot of the best practices and knowledge we have about programming comes from organizations that are building new applications or at least very young applications right like and especially like consumer applications all the social media applications and things like that versus like one of the things that that's different at a large older organization is like you're saying or we're saying is they have a 30 year old application

and just migrating all those business rules is going to take a long time and so you have to have this realistic expectation of like yeah it's going to take like two years and that's not bad it's fine and in fact it's good to be slow and steady and careful when you're doing this because it's important right like it's not like i don't know i'm just making something up if you're doing like a photo management app like it is a bummer if you lose someone's photos but it's not that big of a deal in the grand scheme of the thing where like for these kind of applications across like you know banks and governments and other things like that like it is a big deal if you if you like mess up the business rules like it causes like the scale of problems it can cause is much more than like yeah we lost photos from when my kid was six which is a big bummer for me personally but it doesn't like bring down the banking system yeah you're right you're right that was uh that was indeed the case that was working on that kind of modernization is always sensitive i would say like uh compared to more classical legacy system uh right because they are yeah they are we are touching the core of the of the company but yeah like for example for the concerning this example like it was very important to do it because they outsourced a lot of things and and right they still did they were not controlling this corn anymore now uh but that's it it's a long journey so then finally so like uh well as as i always joke it worked otherwise you probably wouldn't have written it up like like what were the results the outcomes at the end a big outcome but that is not really technical it's more like the trust like people people are not sure that it will work often we are used to to try to anticipate a lot of things uh beforehand and and that can be that can feel very scary to try to discover things uh little by little and solve

problems as soon as when when they arrive rather than trying to solve problems that maybe we will not have and and maybe that yeah that so so building this trust this uh um giving confidence that our approach can help them modernizing uh demand frames was one of the big outcomes uh for this engagement another one was like uh also uh but we also have in other engagements not specific to mainframe but mainframe but uh the capability to deploy new features very fast like building a pipeline and and being able to to deploy new features without downtime and stuff like that that was a key learning for them of course uh our practices they left like uh being able to to pair to do tdd and and uh having a lean approach to order the way they are working was also very very key to come back to the pipeline and to give some numbers that i have in mind i think uh before like i said that it was taking weekends to deploy new features uh now you just take 30 minutes like to deploy a new feature so that's that's a big huge that's basically uh the bigger comes that we had you know i i don't really know what sociotechnical systems means i should learn that phrase one day i think i think as with all academic things it's probably too precise for me to use correctly and and fully know one of the things that you know you've gotten over several times is like when you're when you're looking at modernizing something like this you have to not only think about the application or even like no this the hardware system or whatever involved but you've got to like treat the whole organization as the application right the way that people work together the knowledge that they have and then and then also like it's well it's easy to say and maybe a little weird to think of but like your your trust and confidence in the process is part of it as well right like it's almost like a form of qa or quality that like you have to maintain

confidence in it and and instill that before you people can execute all right and uh yeah regarding also the you mentioned the the and indirectly the conway conway's law like about the organization communication structures yeah that was also uh something that we bring uh like uh building these teams that who has uh don't know the capability to take decisions regarding where they want to go to implement the things that was also something uh uh very a bit new for the for the team because there uh it's hard it can be hard when you you you have said you are responsible of the decision you are going to take as a team uh it's not someone else who will take will explain to you what you have to do anymore uh and and building that kind of mindset also is a it's part of the of the engagement it was uh it's very important to to to instill uh to to adopt another culture let's say you know yeah such companies yeah yeah well great well i that was fun talking through that and i think um you know i'll definitely put a link to the blog overview overview the the blog uh which which i think like it'll definitely it has a lot of visuals about each step uh that that y'all went through and i think uh i think it'll help people flesh out the details there do you do you have any any like uh any websites or places you would send people if they're interested in uh following you seeing what else you're up to yeah there's my blog uh my blog i had in a little bit but i i'm going to do that soon yeah so my blog for them d.com basically but it's in french i might try to post some english articles as well well good well i'll i'll put a link to that as well and uh yeah it was it was it was fun uh talking with you about this as always this has been tanzu talk if you're listening to the podcast version of this and you want to get the the show notes you can go to tanzu talk.com and uh find them and hopefully if you've listened this far and you don't subscribe to the podcast you should definitely subscribe for for interviews and news in the space and things like that and you can also find

out how to see watch the videos of interviews that we do uh for things like this if you go to tanzu talk.com videos and with that we'll see everyone next time bye bye

Tanzu Talk · 🍕 Get more pizza.