Lessons from 5 years of agile transformation at Jaguar Land Rover

Episode 273 · Tue, 15 Aug 2023 15:27:11 +0200 · 59:40 · SRT · archive.org · all Tanzu Talk

well uh thanks to brian we have we have a a guest who actually works in the you know i i don't know brian it can be said we work in the real world but you know who someone who works in the real world not the talking with everyone in a consultative way finding out what goes on in the real world which which i always uh find fun we speak we we anyways i could go on but i've dug myself into a strange rhetorical trap here why don't you introduce yourself guest oh yeah thanks having me so i'm mike scaith i'm head of engineering at jlr in the uk as i don't know um so i'm at a place called the digital delivery center in manchester in the uk so we essentially are jlr's digital software engineering devops hub up here in the northwest of the uk uh been here for about five years or so um building various software systems with jlr's i'm sure we'll talk about in this session uh my background personally so i started out as a software engineer different types routines of architecture technical leadership now being sort of head of engineering here so it was a brand new um brand new team brand new capability for jlr five years ago and we built it out now to being here 100 or so different engineers dev tests sre platform security etc uh yeah doing some hopefully some interesting stuff uh yeah and and so so it sounds like uh you know you're you're you're in a part of the company i mean exactly what i'm always interested in where you write the software i mean you and the people that you work with your group you you write the software the custom software that uh that jlr uses right i mean which which is that's great you went from programmer to uh you know managing other programmers and and the bit the big thinking and things like that which is always a topic on its own so so the first thing i i was i was interested in talking about is um you know i've i've been around long enough to know at least that i have this kind of like uh naive understanding of of the software that companies do like it's especially apparent in banking where like in in finance there's all sorts of stuff they do but my view of banking is just like i log in

and look at my accounts and pay bills and so like anytime i think about software it's always that and i talk with uh you know some of your peers brian in in your group and they'll be like Coté Coté that's such a small part of banking there's so many other things that that go so like what is like when when y'all are working on like the software that jlr uses like what is the software like what does it do how does it fit into what the business needs yeah good question i'll give you an overview of of all the different software in general and i'll talk about the bit that we specifically work on i guess so i mean obviously software is everywhere these days in the automotive um i think the car itself um is with software so the most obvious part there is what we call the infotainment system that sort of screen on your dashboard when you sat now apple carplay etc um behind the scenes on the vehicle if you like there's obviously a ton of other software so i think there's something like close to a hundred different what's called ecus or electronic control units on the vehicle doing all sorts of you know braking systems engine management autonomous driving cruise control lights brakes everything's kind of software control these days right a ton of stuff on there and actually kind of yeah you're embedded software you see you see the real hardcore um around the vehicle so you've got um mobile apps so if you buy a fancy jaguar or a land driver you get a mobile app with the car that lets you use various connected services so you can you know lock and unlock your vehicle remotely you can set the temperature while you're having your breakfast and warm or cool warm up or cool down your car while you're eating breakfast you know important stuff like that um and then you've got kind of yeah various systems i guess customer facing web systems so obviously you can buy a vehicle online you can configure a vehicle by like your finance tool or that sort of stuff with it we've got you know a service where you can buy a vehicle on a subscription basis for example um there's various kind of again connectivity around yeah connected services like um the app i talked about there's various apis around the vehicle that can sort of do various vehicle we can send software updates to a vehicle which is one of the areas we work on

so i'll talk about a second but then there's other stuff you don't think about kind of scenes again so like manufacturing plants again all software driven right robots manufacturing systems all the scheduling of that massively complex or software driven um other kind of business hr systems behind the scenes as well um design design user software yeah vehicle testing increasingly uses software so rather than smashing physical cars into a wall we try and use some software to do that and visualize that seems like an improvement that's fine but improvement less expensive yeah so it's a deadware essentially right and there's sort of like a finance systems or sort of usual business software and so the bit where my team specifically fit in so essentially what's called off-board web systems essentially so uh we're not doing the vehicle the software on the vehicle itself we'll do the stuff around it so we do a lot of apis and connectivity so that mobile app that i mentioned we do the um manage the subscriptions and the apis that provision those subscriptions on the vehicle um you manage that system that allows you to buy a vehicle subscription so rather than you know purchase or lease a vehicle you can sort of pay a fix and three but kind of all in deal with insurance breakdown etc you can manage that um yeah big enterprise api platform uh and then something called sota which is software over the air so that's a software platform for sending software to a vehicle bit method um but we're involved in that sort of things as well uh and yeah various sorts of web web systems like that essentially so web apps cloud that's our sort of sweet spot yeah and then yes again we're not the only engineering team in jail of course there's other teams also there's teams building that onboard software itself on vehicles most of that's done in-house some of it's done by partners for example uh but yeah that's that's kind of the sweet spot where we think yeah you know it's it's it's uh uh i'm i'm always as as a connoisseur i was joking earlier about the the fun world that we live in like i'm i'm always curious about like what digital means right it means different things in different domains but as you're going through like like what your group works on it seems like

there's kind of two things one like digital means like you know imagine things that are sort of like let's say manual i guess that's the opposite of digital or analog right but like kind of analog manual things and imagine like replacing them with software and then on top of that like if we do that like you can think about the different sorts of new things you could do like like you know for example you might have like i'm again with my layman thing of thinking of things you might have like power windows right which is a very analog manual motor to put in somewhere but if if you made those more like digital and software driven you might i don't know this is a dumb example but now these power windows can do a dance right like you could actually send the software over the wire to do things some more complicated which which i think is like you know that's one of the more interesting i don't know maybe the most interesting like meaning of digital is like we're using like the fungible agile nature of software to like change how things work right beyond just like i can make this character bold right that to actually like change day-to-day activities and and make them more interesting exactly yeah yeah we we use software to improve the vehicle over its lifetime it used to be in the past when you'd buy a car and that was it you're stuck with it for life but we can now use digital and software to get to actually improve that car over its lifetime the best example of that one is where um our first electric vehicle the jaguar ipace um we've got a racing team in formula e electric racing series so that's all sort of software based as well and the racing team there made some software tree tweaks on the racing car to essentially extend the battery life or improve the efficiency efficiency battery and we were able to then translate that change to the systems on the ipace road car and send the software update over the air to essentially improve the battery efficiency of a road car after someone had bought it which is pretty cool right yeah

yeah or or like living in amsterdam the car i have now uh i just need to get the uh the english version of everything on the dashboard then that would be nice to get over the air i mean just a very a very simple thing well that's so it's about to see it so we can we can sort of which on and off features remotely as well right so a good scenario there is imagine you've got a vehicle rental company like hertz or avius for example you've got a fleet of jaguar and london vehicle you can configure those per customer so you can sort of enable and disable features charge back to enable and on a heated seats etc um right do that through a software configuration over the air as well so the the power of it is is quite immense the flexibility offers you and the possibilities it offers you going forward is pretty impressive right yeah manufacturer as well right so that you're building the same car or you know many similar components but you're you're just reprogramming the software either at time or or remotely um there's still as well there's a big challenge as well when you're when you think about those electric windows it's not just some software from jaguar that are deciding when you push this button it goes up and down you're going to have components from bosch and other manufacturers tying that all together so you have all the same problems that you'd see like we see in the enterprise of older software talking to newer software apis abstraction layers and everything in between is it the same for jaguar as well uh yes it is it's very very complex the vehicle's massively complex like i say i think there's something like a hundred different ecs in the vehicle or something like that um a lot of them are built in house actually we were quite looking general a lot of it was bought in house a while ago and yeah there is still components from yeah bosch lg other partners etc um and getting them to interact and talk to each other and i think dependency management that's the biggest challenge right so when i talk about sending software updates over the air so say we're going to update one of those modules over the air there's a massively complex you know world web of dependencies and things compatible versions etc that's very very difficult to manage so yeah it's very very complex um to understand which version which ecu works with

other versions etc that's incredible it's not trying to coordinate that in such a way that we don't obviously break someone's vehicle that's the worst case scenario there's a lot of complex dependency management we have what's called a digital twin that we use which is a kind of i guess big database big representation of all the different software on each different vehicle again what version is comparable with which etc so it's a bit of a complex thing it's not like i guess you're just firing software out like you would maybe you know you can't tear it down like a kubernetes pod and spell a new one up a bit more complex than that right yeah that's an interesting point you you raise brian like it's uh it's it's i mean this is a discussion for for another time or or or here or whatever but like it is uh maybe maybe one of the the key defining things of like let's call it enterprise software both like for office work so to speak but also as we're describing here for like like industrial things is is like at some point just managing your dependencies and everything you work with becomes almost a huge part of your job like just like if you want to forward something like it's hard to like you know going back to my now silly analogy you just want to update the power windows and next thing you know you're worrying about like how the tire pressure is checked and like this doesn't really make sense on an ideal whiteboard but at some point it made sense which which becomes complicated and so and so like going like kind of going to a slightly different area but definitely in that area of dependency management right so like i find that in manufacturing of of of any type uh there is like there there's this weird situation that that that comes up in my head so one in manufacturing like i mean it's it's literally like they're in the world with the word you have lean manufacturing which again from my naive understanding is sort of like one of the better ways to do manufacturing versus other ways and a lot of what we've been doing in the past let's see what i always have to remember

what what year this is so that i can modify this but in the past like 20 years or so like we in the software world have been trying i think genuinely sincerely very hard to like learn from lean manufacturing and and and the way that it works and you know i'm often curious in manufacturing organizations like how that's going for the software people right like how the application of lean sync think uh kind of manifests itself in the software world because in theory people working in software manufacturing companies would be familiar with it and you know you might just run to someone in the uh the cafeteria who's like oh yes i'm a triple judicious black belt of whatever so i'm here to answer your questions about it but like but like how how how do you apply like lean think to the software process good question for you yeah and they are two very different worlds inside jlr specifically i think if you go to one of the manufacturing plants and you haven't got a chance to do that it's quite mind-blowing the automation there and the process there it's incredible to see in action um and yeah some of the some of the cultural aspects as well so there's something called the and on cord where you can pull this cord on the manufacturing line anyone can pull it if something's going wrong everyone sort of mobs around an issue review it have a post-mortem if you like after the incident and all that sort of cultural stuff is there in the manufacturing plants but in the digital world the off-board software world before we arrived a lot of that wasn't in place it was very old school if you like it was you had waterfall development not agile etc it's kind of weird it's kind of weird how you've got those two worlds because like i said there were plenty of people in the business who understand those principles of lead maybe haven't related them to devops and i don't specifically they understand the culture side of it and understand how that benefits um well yeah they are very two very different worlds i think um i guess that that's one of the roles that our team's played here has been trying to sort of not force but yeah sort of drive some of that transformation to

those similar ways of working some of that similar culture in the digital as well um and it's tricky so so again i don't know if this is jlr specific thing we've got a lot of people in digital who you know have been here for 10 15 20 30 plus years they've never really known anything other than jlr software development so there's been a bit of an education piece there um then there's sort of actual transformation work as well so we've tried to introduce introduce things like ci cd that was a whole battle in itself i'm sure we can talk about um agile ways of working again that's all quite new and i'm sure we'll talk about how that ties in with some of the other processes like finance etc the point you know it's very traditional dev support dev test support separate teams separate parts of the process all that needs to change as well um so yeah the the assumption real perspective is right why is the whole business not working in that same way but for whatever reason the software the digital the it side of it was lagging behind and it's still sort of going through that transformation i guess part of my challenge is then yeah is trying to relate what we're talking about in that world to what they know in the manufacturing world and how there's similar similar concepts in the core pathways like i say that's the trip to it really yeah yeah well well brian tell me like like you especially nowadays like like me you talk with a lot of different organizations right and like uh i i wonder like there's that gap between like like you know here lean thinking in the core business versus like i don't even know how to express it but the theory that i have and this is what i i don't know looking for i'm always looking for validation but you know it's it's it's it'd be interesting to kind of like help shape the theory is that i think historically most organizations kind of think of software as like a supplier like like in the same way they might ask for like you know we don't just need like like the standard binder clip to hold our paper together we need this special binder clip but really i'm going

to draw you a picture and i just need that binder clip and just deliver it to me right there's there's not sort of like a more sophisticated understanding you know like a product management approach to binders clips would annoyingly be like well let's step back what are you doing with these binder clips but i i don't know it seems like a lot of organizations just think of software as like an off-the-shelf thing and and therefore i don't know they don't kind of craft this more agile way of thinking about it intuitively or something so to to really milk your um your story about power windows um so i always love a good analogy especially when it gets silly um like if you think about a power window like in a modern car as mike was saying you've got an ecu that's connected to computer there's dependencies software programmed it could be put in many different cars but there'll be an engineer somewhere that just says why can't we just have a switch connect to a motor that just drives that it'll be nice and simple it'll be straightforward it'll be easy to easier to fix easier to to work with easier to design perhaps and it's only when you look at the the full picture of well actually that means we've got extra wires running into that door frame that means that we've got more things that can physically break it means that we've got a different switch in different cars so it affects the manufacturing process i think software is kind of the same you need to have an appreciation for how software can fit together ideally in a business whereas i think because we use it every day in our lives it's very straightforward thing well we'll we'll just make it do this or we'll just do that i think that's the difference i think businesses especially say the last five years have discovered that developing their own software which on face value seems really complicated right there's massive value there because i think in the manufacturing industry specifically around cars like it used to be we talked about zero to 60 and engine sizes and

literate and all that whereas now we're talking a lot more about the the technology that's in cars um you know because i'm working with some some other old moves as well a lot of my social media feeds and research is full of the younger buyers not that i'm calling myself old quite yet but the younger buyers that are now looking at cars and they're really upset that the navigation software is not as good as what's on their phone right so i think challenge of understanding that we can make new value through software but appreciating that means that what we're going to build might be more complicated and there's more step and i think yeah carrying people through that process might challenging because we know a different way of doing it right yeah yeah yeah i get i guess like maybe the way i'd clarify my my foggy thinking when when i when i was talking earlier is like i guess i guess from my perspective and maybe from the perspective of the three of us a lot of organizations think of software as just like some commodity thing like it's not it's just like a supply that you buy it's not something that you want to customize and specialize because you can to really like be part of the product that you're building just exactly you're saying is like you know a really good like software driven dashboard like including the navigation and all this stuff is actually like a really important feature like it's it's a new feature that didn't exist that becomes part of your product so and maybe that's that's where like i feel like a lot of the consternating if that's a noun you can make into a verb like like happens is that like all of a sudden if you as an organization want to treat software as something other than a commodity you've got to give it the same attention that you would the other parts of your core business and then as you're getting to uh brian and then you discover the complexity of software

right which which was easy to kind of hold if you you know hold back if you didn't want to customize it uh very much and transforming people right that's always the most interesting part of all of these journeys is that we can bring you software we can develop all these things we could hire lots of people but we need to take everyone on that journey um and they're all going to do that at different speeds universal across all the businesses i've worked with um i'm you know i generally say people say well i'm working with technology don't a lot of stuff i do day to day is actually just working with people um and helping them understand what that might look like and be more empathic in the way that we approach that change yeah yeah so so uh you you can help us get or at least me get a little grounded here mike i've i've stuck my head way up into the the the metaphor for like wacky zone uh but as you mentioned right like like i guess about five years ago you were moving from a uh you know a more waterfall way of doing things right which which i'll presume means there's a lot of upfront planning like at least on a yearly if not multi-year basis you specify what you want on day one and if you're lucky you get exactly what you asked for on day 365 right which which you know the clever way of phrasing that is like maybe you don't want it anymore but whatever right like so you have that kind of long process um so what were like like like what was first like the in-state goal that you were looking for with being more agile and like how did you sort of like like what's the broad journey that it took that it took you to get there and then maybe we can hop into like the little obstacles and you know stories of of each yeah good question and brian's right here the majority of it is people based as well we'll say that the software is the easy bit right um yeah of course so five years ago yeah so you're quite right because a very waterfall um 12 months funding to do a project then it's done that's it give me some

requirements documents see you in six months time when it's finished yeah that's the classic world right that's what we walked into um in terms of end goal i think it's probably a couple of things so one of it was around quality of software so making sure you have a real focus like we were just saying a second ago with sap are becoming more and more a crucial part of business start treating it the same way we do as it's quality on our vehicles you know we wouldn't send a half built vehicle out into the world but break them all the time but have the same attitude towards the software so quality was one part of it and i think speed was the other one as well right so again because software is now everywhere because software as you just touched on there is differentiated these days but to make sure that you know we can respond to change quicker changing customer demands that example i gave before about the sort of extending the back you know being able to sort of make sure we can respond to those sort of things a lot quicker so so speed and quality i think probably the two two main motivators there's probably a cost thing in there as well right but yeah um yeah probably those two things so in terms of the journey um one of the first things we went after in terms of bottlenecks i suppose what's in the bottlenecks in the process that are stopping that speed at the moment the main one being essentially getting things into production getting things released that kind of yeah that release cycle that lead time i recall um so we went after what jlr calls our change management process which is essentially to get any software change into a productive environment there's a sort of two week very governance heavy process where you've got to suddenly raise a ticket in a system it gets reviewed and approved by 87 different people 86 of you don't have any understanding what the change even is um take the wages to get through that cycle you've got to agree kind of downtime over a weekend what's your rollback plan but it's obviously the exact opposite of DevOps right so that's the first thing we went after was how do we if it's going to take two weeks to do a production release it's going to sort of kill a new attempt we've got to do is add to our ways of working etc make sure we can sort of do it multiple times per day so that's the first thing we went after and again the majority of that really was people

side of things it was convincing people about how this fancy new way of working we were talking about just as good if not better than the original process how we'd still address the risk in the same way how it was how it was even less risky to do smaller more frequent deployments or good stuff using automation etc all the vendor reviews etc that's the first thing when we went to that was the biggest thing the biggest win because that obviously that unlocks everything else right once you can start pushing things to production no you want it unlocks everything else building in quality responding to demand etc etc uh-huh yes i'm rumbling on a bit there but i think yeah that's that's probably the biggest we went after and that helps improve speed improve the quality and we're built on that foundation from there essentially and by doing that by sort of having that first use case and started to push out some of our first few releases we then earn that trust to people to say actually okay these people know what they're talking about um and from there we've been able to build on on the sort of further change that that resonates with you brian but that's that's the main thing right now it's a great way as well of showing intent right so people can see that you've made a real world difference in how quickly you can do something all of a sudden your the rest of your plans might not seem so crazy yep yeah and and well so so i'm curious like like a lot of what you mentioned was um well not a lot of what you mentioned you most of it was just releasing the software right was was difficulty and you know does that does that imply that the actual like programming was kind of fine or or did the the like application developers did they also like need to change how how they were doing things okay i mean there's there's stuff around that of course that enables that automated um deployment so far so we're talking about things like um automated testing that wasn't in place before either um all right all that sort of stuff right yeah it's just stuff that enables the devops release cycle so yeah ultimate testing the crcd pipelines all the infrastructures kind of stuff that wasn't there as well so it's all that other other stuff

that underpins the release process if you like the end goal was the faster release cycle and yeah to your point got here yeah there's some some changes in the way you actually build the software how we do the test and the fact that we all do that together in one team it's not handed off built tested by someone else or by something else it's all done with that one team there's that culture well around yeah the building of the software the end goal being release anytime so so how much like well at least at least you know in general but in your direct experience how much of the way people work day to day whether it's in development or operations or like architecture i mean all the various roles that are associated with getting software out the door how much do you think is driven by that pipeline and and not driven as in the pipeline does it but driven by the needs of the pipeline right like like analogously like you can look at the way a city is designed and you can say like ah the city is designed around the technology of bicycles or the city is designed around the technology of cars right and and you're making me realize i've always been curious like how the tool drives the culture if if if that makes sense and so like you have the goal i want to put this the cicd pipeline in place and therefore not only can we change things like having less manual reviews which is great but we also need to change these things to take full advantage of it yeah definitely we think that's the approach we took is we tried to use that to force the change out not force a change way to drive the change elsewhere right to reveal the benefits of changing yeah or whatever they were it became apparent we couldn't have a separate test team we couldn't do manual testing we couldn't have this um process yeah yeah exactly that i don't know if we consciously did that but once we started talking about the need to deploy much more frequently it kind of unearthed all these other blockers we did to sort of work on as well that makes sense i didn't remember the answer yeah yeah um i've lost my show i thought i'd be okay yeah well like for

example like along along the area there that i was asking on it seems like like like like one consequence for application developers is if you are if you're going to be releasing your software every week instead of every as as let's say every month i mean if you have your final release every six months probably in development you're at least targeting like a month to kind of like bring all your code together whatever right you've got your own internal things but once you reduce your release cycle to like a week um because you want to have you want to support like the cicd doing this like like like it's it's obvious but it has kind of profound consequences like you can do less each release right like so you have to force yourself to do less code which sort of forces you as the way you're operating as an application developer to think in smaller chunks of functionality i don't know it has all sorts of consequences right and similarly like like i would imagine in operations like if you're supporting a pipeline like the way you do service request management has to change i don't i don't even know right but but it's it's interesting to think about like what kind of constraints or what kind of motivations that you get when you just because you're using a cicd pipeline right yeah even like architecture for example right so beforehand there was massive monolithic applications now we've moved to a more macrosix architecture to change things more independently all that sort of stuff and great all that sort of stuff but and like i say that's mentioned automated testing that's the same thing if you're going to release it quickly you need to be able to test it much quickly so we can't suddenly have big manual test scripts that we can work through step by step it's got to be automated and much more quickly all that sort of stuff comes off the back of it you're quite right um yeah the service misery piece is another interesting one because again there's uh this is where perhaps we're a little bit separated from other parts of jlr but jlr's got its sort of traditional it help desk and we kind of but that's very much um yeah ticket driven manual driven again and we kind of feed the hang off the

back of that with our instant process so that that's my next area to go after and change actually is how we can sort of integrate more closer with that and yeah you're quite right coach it's it's it's driven some changes in in the way that we work in terms of building the software but the way that our stakeholders that we work with think about it as well right so like i said they can't suddenly come along with a big requirements document and say see you in six months we need that right as well right it's got to be we need to talk to you on an almost daily basis as well and get your feedback and make sure you're making the changes in the right way the people side of it as well as the tech side changes off the back of that that need to release quickly to production which is why that was the first one that's the enabler for everything else i think is getting stuff out yeah so so you brought up you brought up a a situation that i'm realizing connects to like this this thing i've been uh not stressing out but like you know staring at the wall and thinking about like so so the one you brought up was that to to to phrase it again in an experimental way so we wanted we for reasons of quality and also for like speed of getting features out so that so you have time to market basically um and and then also what comes with that is like we can experiments always about a scary word in manufacturing but we can try new things out business wise and see if it's a good business or not right we're not committed to like 10 years of something because we decided this thing in the physical whatever so but you can kind of refine and try out new business ideas um so you put a cicd pipeline in place so you can deliver things quickly um and as a consequence to that as you were saying like the way you architect your software has to be different and like microservices ideas are a good fit for that because microservices is basically like it's kind of like to not go on and on about it it's like having agility in your architecture by removing dependencies and kind of having smaller things um now with that same kind of like it's almost like a uh what's the the idiom a cart and a horse

or a mule like which one's leading which and like if i look over in the infrastructure space i'm often wondering like i mean here we are at vmware we're big cloud native proponents but i often wonder like can you do agile development because you now have cloud native infrastructure or do you only do cloud native infrastructure because you want to do agile development right like like what where does it come in first and so like to that and i'm curious like when you move to this like shorter cycle like what did it uncover about the way you ran your infrastructure and you did your release management and all that operation stuff like what did it require you to do yeah i mean it included a lot of stuff right because i've mentioned things like testing there i mentioned the change render a bit there's then things like security as well and that's one of the part we've not talked on yet and then the infrastructure side of things um yeah i mean i think when we when we first wrapped up five years ago it was the beginnings of some cloud infrastructure there um jc but again to get that that involved raising a ticket against the team they'd spin some vms up etc and obviously we realized that that wasn't going to work for zyber so that that led us to then using things like terraform infrastructure as code and provisioning that in an automated way as well um that's one thing you're going to call it the security thing is important as well because again that's where you've got a traditional view of build the thing do a pen test if you're releasing and again again you need a different approach to security there as well so that's where we start to build in things like um vulnerability management and our pipelines etc how do we do more automated pen testing on a more frequent basis all that things there um then leads to who pays for this stuff we've got to look at furniture i'm sure we'll talk about that in a second that got all covered there as well um but yeah the whole it's interesting what came first agile or what i think they're both kind of hand in hand in that sense we we realized that well we knew from the start that we wanted to work in a certain way right we wanted to do agile wanted to do DevOps and that led to us uncovering all these different parts of jlr that sort of didn't facilitate that at the moment and driving that change

yeah yeah i mean i guess i guess another way of of like thinking about it which which i've been trying to ponder is like why why couldn't you just like use what you already had right like like what like what uh what was preventing you with like just the infrastructure and the process we've kind of covered but like the tools that you already had such that you would want to try out new things good question think about one second you know i i think that's a really good question because i think this is where i've seen a lot of enterprises stumble and i think the answer is everything like if you want to build a car by hand you would approach it you'd approach everything in a very certain way whereas if you want to build 10 000 cars in a factory in a manufacturing line your thinking is different your culture is different your design is different your parts and management supply chain is going to be different the way you put them together is going to be different you're not going to start at the bottom and work your way up you're maybe going to do different panels of 10 cars first you'd be thinking about your release strategy in terms of having batches so that you can you can see what's changed over time so if someone reports a defect in a batch of cars you can then make a change or back to the mics this this concept bikes where you've got this this cable you can pull in the manufacturing process and that's that's got a real bit of psychological safety in in the workplace right you need to be brave enough to stop the manufacturing line to say i think we've got a problem let's all gather around and fix it and i think that's one thing we don't do well enough in software about getting in that tangent just now but i think yeah and sometimes it's where to begin like that's the nice thing about mike's story is that he he found a place to begin and start and then made changes and we're still on that journey right it's going to be you know we're not fixed yet that's why we're still still all in gainful

employment which is lovely for for technology yeah i think it's everything mike is that what would you say yeah i agree i agree i think that's why yeah you're definitely right we're still on this journey right i don't want to paint this picture that jlr is this perfect software hustlers but not um yeah we've got a lot of challenges to fix still um i've mentioned the instant management side of things there yeah i agree brian i think it included everything um and the cultural aspect is really interesting to me let's talk about psychological psychological safety um i believe that's something that we've tried to influence as well right because again there's an element of when we had our first incidents or first few incidents like okay who do i blame who do i suck for this thing but we thought that we're going to take our time it's a blameless culture we're going to find the root cause we're going to review it together and identify how it happens again it's always a system problem not a person problem all that good stuff that's another part of the journey we've not talked about yet where yeah the tech is one part of it and that's fine but the the cultural side of things has been probably a bigger challenge but almost certainly a bigger challenge and try to change people's ways of thinking around uh like say incidents risk security testing all that sort of stuff again yeah i agree it it's just pervasive brian you're right so so there there's there's two two sort of hurdles or bottlenecks that that that i wanted to to talk about before before we run out of time and you you just one of them is uh as as you keep saying finance which which uh which is great because people always tell me finance is a problem and i have no idea why so it's always good to uh to ask people uh but the the other one was um changing how people work or changing the culture right and then and then i think i think you're hitting on a nuance that i don't think about too much is changing their perception which is usually fear about something right that's something their trust about something but but instead of me feeding you ideas as as i as i've kind of been uh allowing myself to do here too much like like what are those culture change problems like what are the what are

the what are some of the the issues that that you faced because i often feel like if i went and read like a book on like just you know generic agile software development we could all agree it sounds great right like we can all agree on this in state that we want to get to but like getting there seems very difficult yeah there's a few cultural elements so i think risk is definitely one of them um it was definitely a kind of real risk aversion jlr around software changes possibly driven by you know being bitten in the past by big releases that have gone wrong etc you know if you're going to do releases all the time that's that's quite scary there's a risk thing there um and of course the impact of some of the software changes as well if you're talking about changing software on a vehicle there's a lot of risk to that right what if that goes wrong and you break someone's car or you know the braking system stops working or you know it can cause a lot of damage for that so i understand the risk aversion in that sense um another interesting part and again i don't know if this is jlr specific is i guess i'd call it siloed ways of working so when we're talking about agile teams working much more closely together end-to-end um when you've gone into world where you've got you know dev is here test is here release is here one of our supports there um there was yeah there's lots of very very siloed teams so we've had to sort of change how those teams work together um we've built kind of cross-functional squads to use the the buzzword um of all those different types of engineers and different business stakeholders working more closely together than they have done before i've mentioned a few times the idea of you know we get a requirements document thrown at us see you in six months that's how to completely change right with the relationship we've had with between the digital teams and the different stakeholders from other parts of the business that have been completely changed because as i said we can't we need to work much more closely together um to your point about being a commodity it's not something we can just check over the wall and get back when it's finished we need to change that kind of operation and those are probably the two biggest things you have that kind of risk aversion and then the collaboration side of things it may be specific to dll because jlr is just such a massive

organization with so many different silos but i'm sure plenty of other people out there large enterprise companies see the same challenges right you're not alone you're not alone how about you brian like like what do you see as like like like if we go like one or two as as mike was doing there if we go one or two layers below like culture change is hard like like not only what but like why is it hard like like like to pick one thing that i think is good with the perspective you have looking at as you were just you know uh proving many different organizations right like like so if you have if you're changing the way that groups interact with each other or silos right like again it seems like we can all agree that it would be better if we didn't have silos and we all work together and yet like it's such a difficult thing to get people to change and so like what is i don't know why is that why why is the status quo so powerful so have you spent a lot of time thinking about this because like change is really really hard i mean we talk about it we all just say change is hard and trying to build a structure around it is is where i'm spending a lot of time so i think there's a couple things i think first of all um been writing article recently that you might recognize the subject of but bottom up and top down change um so you did a great video series around bottlenecks and i've been writing up some of the um the blog post on it writing on cocktails what i do best um but in that i've spent a lot of time really looking at what drives top down and bottom up change to fail or succeed because i've seen both and i've tried both um so i think pop down change we're all we all recognize mostly as a marketing campaign so leadership decide we're going to have a radical change towards agile we're all going to go and lean training we're all going to do this we're all in scrum teams now and you're all going to love it no matter

what and that doesn't work and then likewise bottom up change it hits a limit of local optimization because as you were saying mike like you can you can build a fantastic system to to take an artifact and put it into production but unless qa are on board or less security on board or unless these other teams are on board you're going to be really limited in what you can achieve and so i think the reason that change is hard is because you need a mix of that top down and bottom up and they need to marry up in the middle and that's not something i think many businesses agree at i think we do one or color we don't normally see those two things come together because i don't think there's often a strategy up front to do that so i think when i'm thinking about my experience building platforms and helping customers with product i spend a lot of time talking about advocates and i think that's really important but you need to have people on the ground are going to be advocates for what you're doing why this change is important and if we think about the bell curve of adoption for change right the uh diffusion of innovation that's right oh yeah um we'll see the beginning curve innovators they just love new stuff so they'll do they'll do anyway you've got this other huge section though which are basically pragmatists they are going to adopt change as long as they can see value from it and then unfortunately you've got the other half of the bell curve which are people that are going to be resistant because they know what they're doing just now and the change is really uncomfortable and let's be honest it's uncomfortable for all of us but it's just whether you can see a value of it and i think that's what we need to do with change we need to find a way of getting that first half of the curve on board and then helping the second half but it's part technology it's part process it's part people again change is hard because change is everything um so again that's a rambly need to put a blog post together mindset but i think that that's that's kind of what i see in the real world is that we

just fail to meet in the middle sometimes yeah yeah and and and you you talked about this a few times mike but there's the there's the needing to prove that it's a good idea beyond an intuitive understanding of like uh you know you have to prove that it's a good idea because essentially you have to like prove that it's worth the risk of it not working right and all the effort you're going to put in the position you're going to be in which which i think is uh i mean it's like all common sense it sounds really obvious when you say it but it's it's hard to actually put into practice but yeah speaking of like so again the last topic speaking of resistance to change so what is what is the deal with finance just like give us give us an overview what's how how does finance help how does it how is it a bottleneck that you know like what uh what's what's the mismatch such that like i don't think i've ever heard anyone say we were trying to move to speed up our software development process and boy finance was a big help no it looks okay so it's one of my finance colleagues may say this um yeah so i think the biggest challenge i think i've seen with it is when i think about what is is that classic project versus product mindset finance wants a funding for six months it gets built bosch done whereas obviously the world we want to live in is much more kind of ongoing funding for products um and part of that i think is is not necessarily knowing what you're going to get at the end of it so again if we move away from that world where we find set requirements here's my outcome here's how much it's going to cost done to a world of oh we're agile we don't necessarily know what you're going to get at the end of it if i'm a finance person i'm going to think right you want me to give millions of pounds and i don't know what i'm going to get back at the end of it so i think it all tied back to that for me i think project versus product mindset and that then links up to other things like um we talked about platforms a little bit so the funding of a platform to underpin some of this stuff is very difficult from a finance perspective and if it's not tied to a specific project a specific piece of work with the kind of very specific business outcome it can be different rate buying for it right so

that and that then talks to other things like you know addressing technical data addressing kind of ongoing maintenance of products etc again unless you've got a specific stream of funding it's difficult to sort of get that paid well so i don't get tired about that that's what i've seen in jlr certainly is if it's not tied to a specific project and it's difficult to kind of access that funding and then that tends to you know funding resourcing funding software tooling etc all that stuff has to be tied back to the project so i wonder if that kind of underpins a lot of it just switching that mindset again to being much more long-term product focused and again tied into risk right so the risk of you might end up spending a lot of money and actually get no benefit from it but you you'll learn something along the way but to a finance person of course that's quite scary because that's just essentially throwing money away and and so so how do you like how do you maybe my understanding of the following is incorrect but how do you like square that with like well we just kind of kind of got to like mess around and discover new things about cars right like like we don't really know what's going to result but like we'll discover kind of like what you're saying right like like if if you were well i don't know maybe tell me if i'm being too naive here but if you're if you're funding the uh what was the electric sports car the e2 like racing like like if you're funding that and then like you couldn't have really tied that to a project of discovering a better way to run the car it just sort of happens and you sort of know that like well we're in the business of cars so if we spend a lot of time just driving cars and observe what happens we'll discover things now what that results and i have no idea but like it's will be something so it seems like that like finance is okay with that maybe but but not with kind of what you're saying is like we're going to build a platform we don't really know what we're going to need to build but we're going to need to build it

so could i just have some money for that you're right because it's the same as i probably talked before about how manufacturing and software are different but you know they're using the same principles if you think about a vehicle you build prototypes we design prototypes they sometimes don't go anywhere we throw money after that sort of thing yeah it's the same sort of concept again it's i'm not quite sure why that doesn't translate into the digital software world in quite the same way you know you could spend millions and millions on a prototype vehicle and it never actually get released eventually you might discover there's no market for it or i think there's some engineering challenge along the way it's the same sort of concept again for whatever reason that just doesn't translate into the digital world in the same way we've got that same challenge there to sort of yeah you use it as an analogy i really like that analogy of sort of yeah relating back to lean manufacturing and using that maybe we need to use a kind of vehicle prototype thing as an analogy for the financial use case as well i think one of the things i've seen time and time again is that the the divide in finance and in software delivery is a lack of understanding i don't think that finance really understand it how it's built why it's there sometimes but specifically the mechanism of how we go about building it so how you build a car with a prototype etc there's maybe a a closer understanding um and as software people i think we're terrible at understanding finance absolutely you know so the one of the best things i ever did was when i became a new manager of an engineering team and i realized how out of my depth i was i called up one of the the young finance guys that was helping me with various questions and i said look can i can i just sit down with you for an hour over coffee and you can give me like a five-year-old's guide to finance in this company and i'll give you a five-year-old's guide to cloud computing and why why that exists what platform as a service is what platform product is what we're trying to achieve one of the best hours i've had i learned so much and we continued doing that in fact um on a kind of monthly cadence and it was really useful and for me it was it was the

ability to understand why finance perhaps have some struggles with software teams a couple of big ones that stand out to me number one is that we don't spend the money that we say we're going to spend we either spend much more or much less and ironically you always understand that spending less is actually really bad too because if we've spent less money than we planned to it's not like we we hand it back like a savings account the problem is that means that other parts of the business didn't get to do what they wanted to do and the finance person now has a problem because now he's got a lot of money but we said our stock price was going to be this and the stock market likes repeatability so if we're not on that line that's a big problem so there's a constraint on finance that sometimes the software on we're unaware of i think the other aspect is that we are terrible absolutely terrible at telling finance what value we achieved so i i'd love to know what it's like in jlr but many of the businesses i've worked in personally and the ones i'm working with you go to finance almost like the gatekeeper the troll on the bridge right to try and get the money you get your money you run off with it big big huge sack of cash and you promise you're gonna do good stuff in a year's time and then you never really report back but also well like the modeling that you've done is like i need all the money and i'll tell you when i'm done and one of the things i changed in my time was that i began to look at big scary projects like building a new car building a new platform and i said well actually don't give me the money don't give me all the money i'm going to want this amount over the year but i want this amount this quarter this about next quarter and this and so far and if i get point at the end of the prototype stage i might actually come back to you and say i don't want the money or i might come back and say i need more and having that type of conversation flip things for me from begging for cash to actually being at q3 how you know kind of at that scary point for finance people and finance would approach

me and say how much more could you achieve if we gave you some extra money now and could you get it all receipted and invoiced in time for our timelines and that's because truck being built up and understanding but what's it like in in glr mike is it that is there a good system for reporting back to finance or are they so busy with just trying to guard the cash in the first place that there's never really that that cycle that happens yep it's it's option b brian sadly oh yeah i think we're kind of missing that feedback that's not the trigger um yeah we're in that first situation you talked about where we take the cash run away and we're very very bad at reporting back about what we've done back of it um or at least talking to the finance people very good at kind of talking to business stakeholders about sort of value you've delivered them what the product does but closing that feedback loop with financing it's definitely a big gap um yeah to exactly what we just talked about then right that's it so what we want to get to is exactly that kind of quarterly model just don't give us it all up front just do it bit by bit do the feedback lead review do we need to keep going do we need to do we need to can this project have those discussions not there yet that's the journey you want to go on and i think that will help a lot yeah it's currently the situation where we don't have that feedback loop at the other end to sort of close the loop and say yeah here's what we did here's what we didn't do give us some more money or don't and it's a tremendous power out for finance because all of a sudden you've turned that agile thing we're talking about being quite scary into being a fantastic tool because now finance have an understanding that they don't have to spend all the money they'll still get some value delivered along the way they can pause a project they can ramp up a project all based on those outputs so if you're doing really well and they've got spare money great let's double down well actually you know financial world has changed some something else has happened either macro or in the business itself we need to change our plans agile can help us do that modern cloud native practices can help us do that why monoliths are have their place but they're they're harder in that regard um but i think having that feedback would be so we go building the trust so

we can actually demonstrate that this crazy world of we don't know what we're going to have at the end of it is better than waiting until the end and maybe building their own thing yeah you know y'all are like i i i i like the the points you're making and one of them so so so to add my my my bit to it it seems like like finance maybe in respect to what we're talking about wants three things they want like to be as deterministic as possible right like you know i mean that's the way we all are with money if if i if i put you know 50 50 pounds over here i would love it if one it stays there and doesn't disappear and then two like it gives me a return that's known like in the direction that i wanted right like and in corporate finance it's even more so and then and then two like i think i think brian you're hitting on this is like there's this expectation of repeatability which is almost like being deterministic like you know that this thing is just going to keep acting the same way and returning the same thing and i think that that hits on a little you know it's it's one of the reasons why not spending enough money is annoying because if if there's the opportunity to have made more money you you would have wanted this person to spend more right and and you know kind of have that be repeatable and deterministic you don't want to miss out on making money and then one that i don't think about very much but but but you two were kind of going over is it's almost like it's like two sides of the same thing like finance wants to have or finance would like to have options like optionality when things change they can like put something on pause or spin more which also implies they have controls like they can actually like having an option without being able to do anything about it is annoying it's more stressful right and so they need those controls that they can go in there and then and then the the other side of that which which i'll have to think about some more but i think is

i think it's something that we should work on an industry is not only like speaking the language of finance but like like y'all were saying we should just speak to them like like at the end of the cycle and then also know how to show the the as you were saying uh i think it was you brian the value that got delivered right and so like i think i think intuitively like if we build out a platform and the middleware and all this stuff like we understand the value that was achieved but like it's hard to like you have to express that to people right you have to say that and and it takes a certain amount of of meeting each other in the middle about the technical parts of it right like you've got to get somewhere beyond like i don't know i just bought you a bunch of computers why isn't it working and you have to say like well let me tell you how that works right and and why investing in this cloud native way of doing such as an example was better than this other way of doing it even though it's just computers or something like that but i don't know i i don't know what the uh further delve into that is but i i think i think as y'all two are getting to it's just sort of like uh a misunderstanding not even a misunderstanding just not knowing how each side works and having different expectations for it uh which which i think is helpful well speaking of being helpful we we should uh we should let mike get on to the other helpful things he does rather than entertaining us like it was it was great to go over this stuff it was really nice that uh uh you know brian's been working with you and and he uh he thought to bring you on here so that's good and and i think uh your your willingness to come on is always great because i find that people in your position uh don't always share with the rest of the community what what's going on you know if only because they got other stuff going on but but thanks a lot for going over that that that was that was great if uh is there anything you want to uh if people are interested in more that you would point them to any uh any uh things

like that i mean i'm always happy to talk about any of this sort of stuff devops cloud soft engineering keep a shout out on linkedin i've got no alternative in terms of hiring anybody so it's a great conversation yeah thanks thanks the best of yours look after me good yeah well i appreciate well as always this has been uh tonsu talk you've either watched a video of this or uh you've downloaded a podcast or perhaps one of your your parents or someone in your house is listening to it while they're making a meal and you're thinking what is this you're being subjected to it from uh from the kitchen uh but if you want to get the show notes from this uh we'll put any relevant links you can go to tonsutalk.com and find out how to subscribe to the podcast if you prefer audio find the videos things like that and as always thanks and we'll see everyone next time

Tanzu Talk · 🍕 Get more pizza.