A Developer Perspective on Developer Experience, also, COBOL, with Paul Kelly

Episode 271 · Fri, 28 Jul 2023 11:37:46 +0200 · 58:29 · SRT · archive.org · all Tanzu Talk

Well, hey listeners, something was up with my microphone. So whenever I talk, it's a little clicky. Hopefully that doesn't drive you crazy because this was actually a pretty fun conversation. There are a lot of interesting ideas that came up that we didn't get to explore more, but let's go right into the introduction. Well, we have another interview that you've arranged, Cora. And this one, you know, I was reading the initial blog article about just like the attributes of DevOps stuff. And I got, I got to be honest. I was like, I don't know. It was just, just going to be about, you know, more obvious things that improve developer experience. But as, as I read it, it was actually like a very practical list of just like a punch list. And I mean that in a good way of just like, here's the minimum stuff to do. Like here's actual practical things. Cause a lot of the stuff I've read about DevOps over, DevOps, Dev X over the years and probably heard at DevOps conferences tends to be, um, I don't know how I would characterize it just sort of like, well, that would be great, right? Like just the, uh, the kind of happy land of talking about stuff. But, uh, this article you, you, you sent over was great because it's, uh, actual things that, that you can put into practice. Yeah, absolutely. It's, uh, it's written by Paul Kelly, who is, uh, one of our labs consultants. Our, our, our labs team is a team that goes and works, uh, side by side with developers at different enterprises to help modernize and move their applications to, uh, modern platforms, uh, whether it's tons of application service, formerly cloud foundry or, uh, tons of application platform, a Kubernetes based platform. But if they, uh, this team focuses a lot with on modernizing applications and, um, and through that, of course, they're exposed to a lot of different developer experiences at different enterprises. And so there's that, but talk, Paul also has a very, uh, a long background in technical writing as well as R and D. Um, so really well positioned to understand, uh, developer experience. And this article, uh, I really recommend it. We had a fascinating conversation with Paul. Um, but, uh, the article covers everything from a really solid definition of

developer experience, what it means to the enterprise. Uh, so I think folks from all levels would benefit from reading the article and, uh, and as I mentioned, sort of at the end of the interview, this tension between security and, uh, developer innovation, uh, and, uh, freedom, I guess, and also tips to running your platform as a product so that you can lead towards a, um, a good developer experience and how important all of that is. So it's a great article and we had a great conversation. So yeah, take a listen and, uh, you know, naturally we start by talking about COBOL. Well, I, I wanna, I wanna have you introduce yourself here and a little bit guest, but reading through your background and I was, I was able to, uh, acquire the, the PDF of one of your books on COBOL thanks to the register and a form there. I went to micro focus and I tried to use their Legion form, but an error occurred both in Safari and Chrome. So I know it's not my, my blockers. I haven't gotten the chance to read it, but I was, I was like glancing through it. And, uh, I, I was thinking, you know, uh, you wrote, recently wrote an article on a developer experience, devX, which is great. It was like this list of like, how to make things nice for developers, right? Like almost like a checklist. Uh, but you know, the combination of all that stuff was making me think of like, like, so why did we need more than COBOL? Well, that's a good question, isn't it? Um, cause as old fashioned to be right in as COBOL is, it's, it's got some good points, which is that it's very performant and it's good at handling money, which most languages are not. Um, that's one of my favorite things to be good at. I like, I like that. So, but on the other hand, it is by modern standards, it's not an easy language to work with. Yeah. And, and, and do you think, well, well, first of all, what do you mean by it handles money? Well, that's an intriguing thing. Well, so most languages, their, their default for, for doing any kind of calculation would

be a float or a double. And those are binary types and, um, they don't do money well because money is usually decimal. Yeah. And so you get rounding errors and that becomes particularly significant when you're doing interest calculations. Those errors can mount up really fast. COBOL has, um, has got a kind of decimal type, which is from memory, possibly comp three, but. Was it designed, you think? Was it designed for that purpose? It was designed for that purpose because the first thing it was used for was business applications and what you handle in business, you handle money. Oh, fascinating. So wait, what, how, what, where does your background in COBOL come from? It comes from the many years I worked at Microfocus, um, who were for a long time effectively known as the kind of the COBOL company. They were the, the, the first company to put COBOL actually on, on, uh, microprocessors on PCs. And I mean, they did that even before the IBM PCs. I think they were, I think their first product probably around something like a rare black box. If, uh, if anyone has that long a memory and I, I joined them in the, I guess the, the early nineties. Uh, and at that time their, their product was doing fabulously well because people had business applications and they wanted to run them on microcomputers and micro focus was pretty much the only game in town for that back then. And so, you know, you know, I'm like, like I'm, I'm interested in talking more about like the, the, the world of COBOL because like, I feel like we're in the middle of like, you know, within, in the DevOps world. Like one of my, one of my theories, I don't know if it's like me being snarky or, or not, but like, is like, you know, here we are in 2022 was last year, but now in 2023 and we're sort of like, uh, rediscovering and rebuilding a good developer stack, right?

Like there's a lot of commentary and a lot of products doing that. And so I wonder in, in your like experience since then, and then through like, you know, let's say like the, the, the windows era of things. And then we had like the web and then there was this like rich internet application, like mobile thing. And I don't know what you would call after that where, where we are now cloud maybe, but like, yeah, like, like my, my sense is, and maybe this is just part of being an aging technologist is that like, I don't know, every five years, every five or 10 years, we just like rip everything down to the studs and totally rebuild it. Right. And, and hence the sort of like, why did we need more than COBOL? Right. And I think like, I don't know the COBOL space very well, but I would imagine like a lot of the kind of tools that, that we would be talking about for like developer experience nowadays for whatever type of development you're doing sort of like have existed in multiple times in the COBOL world. Right. To build up all the stuff beyond the languages that you would need and kind of the tools and integrating them together that like you would have that stack. So anyhow, I mean, it does seem like it, it, it does seem like it's, it's interesting, at least to me to think about like, at what point were we like, this thing handles cash really well, but we got to start over. Right. Like we can't, we can't pull it forward, like, and therefore we're going to have to like, get rid of everything and just kind of start from scratch. Cause it feels like we're kind of at the beginning of doing that with a lot of, with like the, we don't even have a name for it anymore, but with the DevX stack of things that we have. Well, it's always struck me that IT has got a lot in common with the fashion industry in the, you know, in the trends, the, the, the, the hemlines go up, the hemlines go down. Everyone gets very excited. Um, but to some extent, the same stuff comes around again and again.

Um, one of the things that kind of, uh, struck people where at the beginning of the web thing, so web one zero was how much it looked like what IBM had built for, um, for mainframe applications, which was Kix, Kix is pseudo conversational. You have the illusion that you're interacting with the computer, but most of the time you're not, you're just sort of terminal and every so often it's going to send something off and you get a reply back. Well, that's exactly how the first web applications worked. They were exciting and Kix was, was dull. So everyone went through a kind of phase of how can we make our Kix applications look like web applications, whether or not that actually made sense. So there's that. Having said that, I guess there, you know, there are genuine improvements that have happened in the technology stack over the years, but I think when you're up close to it, it's hard to see what they are. And it's only when you look back, you think, yeah, that was an improvement or no, that was a really terrible idea. And those two books that you wrote about COBOL, um, they are, so they're written for Java and .NET developers. They were, yeah. And it's, um, and so they provide a guidance on how to, is it, are you actually converting the code? Is it automatically converted from COBOL to Java.NET? So, no. Well, so, um, I mean, I don't want to spend too long to have Microfocus's sort of proprietary technology, but they're probably happy if I do. Um, they have what they call the visual COBOL compiler and that will compile COBOL code, you know, genuine COBOL code written any time in the last, what, 60 years now, and they can compile that to either run on JVM or run on .NET. And then in addition, they added enough object-oriented features that you could actually make that code play nicely with those new languages. So, for people who have business logic, you know, a lot of, have invested a lot of money in, in business logic

written in COBOL, it's a way to pull it forward. Interesting. By recompiling it and then making it talk to newer things. Does that actually make it, uh, like, have you ever seen these COBOL applications then even run in containers and, and run on Kubernetes? Is that a thing? It is a thing. Uh, in the second book that I wrote, I, uh, did a, uh, there, there's a kind of worked example of running a COBOL app in, in Kubernetes. And in fact, one of the things I did when I first started at Pivotal a few years back was I, I did a little demo of running a Spring Boot on, um, the, on Cloud Foundry. Spring Boot with COBOL code. Sorry, Spring Boot with a COBOL. COBOL code. COBOL business logic, but then effectively made that a jar consumed by a Spring Boot front end so that you could just package it and run it in Cloud Foundry. And hey, Presto, you were doing, you know, interest calculations using, um, using COBOL, but running on Cloud Foundry. Wow. You could finally handle money. And you could finally handle money. Because in Java, when you want to handle money, you have to use big decimal and that code is not easy to read. Best of both worlds. Yeah, exactly. So, wait, tell us a little bit about that transition into, um, into Pivotal and this sort of pivot in your own career, I guess, because I guess at that point you became more of a consultant. I did, yeah. I did, yeah. And that was kind of interesting to me because I spent my whole career up to then really in, in effectively R&D, doing package product. And working as a consultant was, was very different in lots of ways. One of the things it, it, it gave me a much better appreciation for the problems that people developing business applications have, because up to then I'd really been doing, I guess you call it more systems programming. But also it did, but also it did, the thing I liked about it was it did give me a chance to go and talk to people a bit, because when you're in R&D, you're kind of in the ivory tower and certainly micro focus.

So, we got to speak to customers once a year at a user conference. Was that also your first experience doing things like paired programming and, and, um, and other such sort of, uh, collaboration techniques that Pivotal employed? We used a bit of pair programming at Microfocus, but it was never kind of part of our core programming model. We used to pair up mainly to solve difficult problems. So, it was a huge shift for you. It was a huge shift in every way. I, I definitely had culture shock for, for several weeks. And do you feel so, so you've written this article, um, so I guess just, you know, so the inspiration for this particular conversation, uh, was rooted in a, in a blog that you wrote. For the VMware Tanzu called the developer perspective on developer experience. Yeah. Um, and so I guess, uh, there's the experience, I guess, working as a consultant doing paired programming and guiding other developers into not improving, not just our code, but also their, their methodologies. Um, as well as I guess this whole revolution that we're experiencing in terms of DevOps and the kinds of automation systems that we are now capable of creating. Um, so I guess, I, I, it seems to me that those two would be the biggest sort of influencers in terms of, uh, tuning in, I guess, to this, to what exactly might make a developer experience good. Or, or is that not, is it broader than that? Well, to be honest, it was broader than that in that, again, when you work at, um, you know, when you work at a software vendor, they treat their programmers really well for obvious reasons. And they make sure they have all the equipment they need to work and that they can find out what they need to know and everything else. Working as a consultant and working particularly with a lot of banks and a lot of telcos, those organizations don't really see themselves in the business of developing software.

But they absolutely are that's now the core of their business. And I think they're kind of waking up to that now, but for a long time, they saw it as just another thing that had to get done. And they don't treat their developers well. Um, you know, the book, the, the, um, the new king makers, that's that, which basically talked about how, uh, the combination of, I guess, public cloud and the explosion of open source software really shifted who had the, the power to sort of crown any kind of technology, uh, as, as more, more successful. And how, you know, it was more, it was increasingly important to make sure that developers were happy, that they were productive. And it feels like, I don't, I don't remember what year that book was written, but I feel like it's at least, could it be 50, 15 years? Something like that. It's a, it's surprising, I guess. I mean, it is, and it isn't surprising, right. To hear that, um, developers are still maybe in some places under a model that, that really like, uh, I think if you're, if you don't see it in person, often, if you're not exposed to it by working with enterprises, it's easy to assume that, that most of the world has, uh, sort of modernized their ways to, to prioritize developer. No, they're a cost center. And that's really the root of the problem. They're a cost center. And even if you can treat them as a cost center, I don't think they, my feeling is that these large enterprises are not thinking about it rationally. So one of the things that I've seen, uh, uh, especially at banks is that they went through a big outsourcing phase back in the nineties and noughties. So they're outsourced a lot of their development and now effectively, some of their development seems to be outsourced to other outsourcers. I'm not sure how it works, but effectively you, you could be at a large retail bank and half their, or maybe more than half their developers not only aren't employed directly by the bank.

Um, they're employed by some kind of third party outsourcer. But that third party outsourcer is the, the person determining what laptop they have and all the rest of it. And their priority presumably is profit. So I have worked, I've paired with developers who have such poor equipment that we are waiting for half an hour for them to boot up their machine and start Eclipse. Wow. Now, in terms of, you know, at the end of it, all the bank is paying for that. The money that's been saved by giving that developer a cheap, crappy computer, I would, I would suggest is, is wasted in, in a few weeks by the amount of time that they are not productive. Wow. One kind of example where this kind of cost center mentality where, well, why would we want to give them nice equipment? It's very short sighted. You're, you're throwing money down the drain. So again, when you're, you know, banks, obviously they can, they can lease their own employees. They could say, well, we'll give you decent laptops, but possibly with their outsourcing contracts, they need to be more specific about not only do you give me, you know, 50 developer bodies to do this work, but I want them equipped in this way. It's, it's ironic because they're, they're trying to do more with less, more, you know, more, more, more, I guess, more work with, with, uh, wherever salaries and costs are lower. Yeah. In reality, they, they're, they're kicking, they're shooting themselves in the foot because they're, they're paying less, but they're also getting less efficiency because developers are blocked. And so developers are blocked, um, communication is difficult. I think, I think you're starting to see, and again, I don't have any statistics or anything to pack this up. This is purely anecdotal, but I've certainly, we, we've had conversations with, with organizations who are now kind of wanting to start bring, bringing more back in house again.

Um, because of the problems of distance, um, time zones, all those things that make, make the outsourcing model difficult. And then on top of that, as I say, you've got this kind of thing, the outsourcers themselves, they want to maximize their profit naturally. Right. You make, and you make it very clear, actually, in your article that the two things that developer experience tight or two of the big, big things that build a good developer experience are tied to is, um, growth for the company and also retention. Yeah. So if you're kind of, you know, if, if working for, you know, even if working as a kind of an outsource contractor for bank X means that you get treated better than the other people in an outsourcing organization, I suspect you'll probably find that the good talent will gravitate towards bank X. It must be, yeah. So what, what, you know, what, what, what we're talking about here is, is, uh, it's making me think like, like, you know, when I was reading through your article, well, of course I was, I was, uh, agreeing and just like, you know, as, as Cora was reacting to like the situation, uh, you're talking about, like, well, I, I think, I think the three of us having kind of the role, like you're saying, we go out and talk with a lot of people. It's, it's, it's, it's still shocking, but not incredibly surprising, right. That, that like, there's this craziness going. So what, one of the things I was wondering and that I struggle with is like, so who needs to hear this, right? Like, like, like who is it in the organizations who's like, oh, right. I never thought of that before or, or something like there's, there's some kind of like disconnect somewhere. Like, like, and just to pile on as, as is my, uh, my habit, like, you know, one of the, one of the things linked off from your article was, uh, uh, like a Forrester consulting thing that, that we have available. And it was, uh, you know, I didn't look up the methodology for the survey, but like, you know, 75% of executives agree that software is important. Right. And so there is like, kind of, as you were saying, everyone agrees with that 2010 software is eating the world thing.

And yet, like, it's to use my favorite kind of gross analogy, right. It's like, everyone says flossing your teeth is good, but probably not everyone does it. So there's like someone in your head that, that hasn't decided to, uh, to, to get a better laptop basically for developers. Yeah. And it's a good question. I mean, I, I kind of, I was hoping this, this particular blog would, would, would reach people in the C-suite because in the end they're the people who make the big decisions, but that kind of, we always blame accountants. Which isn't, I don't think entirely fair because presumably accountants also work within a framework and they have their own goals to hit. Their, their laptop might take 45 minutes to compute an Excel macro. But the point is, whoever it is who's kind of making those budgetary decisions, it needs to be kind of within that framework that says, well, if we save money here, is it costing us money over there? And I, I'd say that it is, but I don't think that those decisions are always taken like that. They're kind of very much a kind of, you know, we, we have a budget of this, this year for, for, for hardware or whatever it happens to be. And we're not going to go over that. I mean, one thing that really struck me as I was reading your article is that the, you know, the term, anytime you have something like a term like developer experience, at least for me, like the, the first time I hear that term, my sort of gut instinct is to go towards the, it's, it's a term that implies sort of a human, it's a human experience. Right. So it's, so there's thoughts that it can elicit about like, are my developers happy? Are they, and it, and, and, and yes, those things might be part of it, but what I really liked about your article and where I think it would be really effective for a C-suite to see it is that it's, it's really, it, it really concentrates on the business decision of it. Right. It's, it is a business decision and developer experience is defined as the, the ease with which software can be created and sent down the workflow all the way to production.

Right. So when people think about DevOps or automation, if they can get it all right, the, the end result of that is developer experience. It's not this other thing that you stick on the left-hand side of it so that, you know, developers are, with a goal of a developer being happy. It's the entire process that already exists in conversations under other terms, being a business, a benefit to your, to your bottom line. If you make those people happy in the first place, they're going to try harder to solve all the other problems for you. And that's what you want. And again, I think there's kind of, I don't know, again, I'm speculating, but some of this is kind of this, this idea that we have a production line and that therefore all the cogs in the machine are exactly the same will perform exactly the same given the same inputs, but people are not like that. And they're not like that to be honest. They're not like that even when they are on an old fashioned production line, assembling automobiles, because, you know, the, I think the, the, the Japanese showed us that having a less kind of, um, us and them kind of split between management and workers made for better productivity and better quality. And I, you know, I think it applies the same, uh, you know, but more so something like, um, software development, which is a kind of intellectual activity where you're being asked to sit and think all day. Right. And, and, and, and by smoothing everything else out, you allow space for that. Yeah. Right. Right. You're hiring talent. Give them, give them an opportunity. Yeah. And don't make it hard for them. Make it, you know, that's the other thing is make the on ramps such as easy for people to do the right thing. Because again, you see this kind of thing again and again, that you, you make it hard for people. So if you make it hard for people to get what they need, they either give up and they don't do anything.

Or if they're motivated, they will find a workaround and now they're not using your guardrails at all. So you've lost control, but you've also wasted a lot of everyone's time. Make, make, make it easy for people to do the right thing. Give them the on ramps to get what they need. Well, well, as you know, you're, you're using the word there and that's, that's one thing that I think you mentioned quite a bit is the, not only like the on ramp to just doing development, but it's almost like the daily on ramp. Yeah. Right. Like the, and you know, I haven't, I haven't read, you know, stories and lore about this kind of thing for, for a while. But I remember when I was a younger developer, right. There was, there was a lot of time being spent talking about this word wasn't used, but as we would say nowadays, like achieving a state of flow. Right. Or I think to use maybe an older word concentration, right. And that, you know, there's all this other stuff associated with what flow is, but just like, you know, you got to sit. And concentrate and, and not be interrupted. And it seems like that seems like one of the more important, impactful things. Like how do you actually get the developer to just start coding, right. And, and maintain their concentration. But also one of the things that like, maybe is very difficult to do and care about. I, I don't know, but what's, what's, what's your thinking about how you like get people just to like, get to the part where they're typing and, and, you know, and to escape from the lines of code, like thing, like, sure. Sometimes not typing is good too, but you know, where you're like solving the problems and working on delivering that business value and stuff. Yeah. That's kind of, that's kind of, that's kind of, that's kind of quite a hard one. And again, the, the, the, the modern kind of desktop environment is not by default conducive to that, because unless you, unless you take the trouble to switch off, you know, email alerts and that thing.

And again, I've paired with so many people, they've got an email alert popping in every 30 seconds and a Teams thing every 60 seconds. So how on earth do you concentrate with all that going on? Like you don't. Yeah. Yeah. I remember early, I get maybe way back in 2015 when I first started working at Pivotal, one of the, exactly this, one of the things that like was a little weird to wrap my head around is like, of course, in, in the, or as we say nowadays, Tanzu Labs, but formerly Pivotal Labs, like they, they would do pair programming on like a shared workstation. And I remember asking like, what's that laptop over on a podium for, and they're like, oh, that's where you can go check email. But there's just like this very much. So a separation of like, here's the coding keyboard and there's the farting around keyboard over there. But it does seem like that. And, and, and, and the, the other thing is like, that's just, it's not even a tip of an iceberg. That's just like a particle floating in the air of like the stack of stuff to worry about. Right. Like it goes on and on and on just to like, focus on like, here's us coding. There's so many other things to, to, to stress out about. Yeah. When you go in for these engagements and you're working with developers, sort of side by side pair programming to either, you know, I guess, move applications, maybe from traditional VMs into some sort of platform. I mean, obviously you're, you're then experiencing the same, either, you know, pain or obstacles that they do when they face the entire automation, I guess, yeah, pipeline that may be available to them. Do you, like, how often do you find that in your role, pair programming with the developers to move applications that you can make a significant impact on the rest of the organization that is responsible for the automation? And I guess, and also, do you see that getting better over time with technology changing?

That's a good question because, I mean, the, the, the sort of the, the labs model of working is that engagement tends to be quite short. So we usually leave a report at the end and we'll say, well, that these are the things you might want to think about if you really want to get stuff done, but we don't always get the chance to go back and, and see how much of an impact it's made. I mean, I think, I think, I think over time, you know, there's a kind of drip drip, maybe they gradually get the message. Well, that's important, right? I mean, just somebody coming in with fresh eyes and, and being able to summarize it and document it and get it into the right hands of the right people should make an impact. You would hope. I mean, at the end of it all, they, they kind of, these organizations pay us because they say they want help to improve their process. So you would hope that they would listen and we say, Hey, there are other things in your process that you haven't even thought about. Right. That haven't, that have ramifications. And they're having a real impact. Right. Right. I mean, you also mentioned in your article, this tension between security and, you know, really letting your developers be a little bit more free and experiment. Yeah. Yeah. And trying to reach a middle ground there. Right. And I guess not, not, not to compromise security and production, but to reach a middle ground. Yeah. And again, you see, you see too often that even in the kind of development platforms, if you want a resource, it's a ticket process. And you not only have to raise a ticket, you have to say what your expected traffic is. Well, I don't know. I haven't built it yet. You know, it's kind of, some of this, some of these processes, they, they kind of, they are, they're very heavy. But anyway, in the design for, we are now, we are now standing up a big heavy duty production system. And you're using that same process for someone who's just trying to get something set up quickly in dev to do a prototype. Yeah. I, I spent, I mean, part of, you know, before I joined Pivotal as well, I had done a lot of consulting and, and hands-on work and, you know, worked for different companies with different kinds of computers.

And I found it always a bit, you know, speaking of security, like, you know, part of my brain under understands security, but part of me was always also very demoralized. Whenever I would encounter a situation where either I didn't have administrative privileges to install something basic, or I would try to help somebody who didn't have administrative privileges. Because I just, I really find it, yeah, it's, it's just demoralizing. It really is. And again, you see it again and again, that they, they're kind of developers on Windows laptops. And either they can't install anything, although that's rarer for developers, but even the developers, they have to go through this kind of tedious process of requesting admin rights that they will then get for 24 hours. So they can install something, you know, it's, and it's demoralizing, especially when you have to do that every day of the week. And I mean, I worked with a, again, with a telco, we were working with people who are being expected to containerize an application. But their laptops weren't really powerful enough to run containers, they were just too slow to run Docker. And then I said, hey, if we could use the Windows system subsystem for Linux, that would be great, because we could install some useful tools. But we couldn't get that working on their laptop, even with the administrative privileges. They'd kind of lock down stuff so that effectively, I don't know if you ever tried to get set up WSL on a Windows machine, but you have to kind of enable some stuff in the OS. And then you have to get Windows distribution from the Windows marketplace. This particular organization, we managed to get through all the early stages, we managed to enable it in the OS. But then you go to the marketplace, and you discover that there's a special lockdown one for the telco. And there's no Linux distribution in there, you can't do it. Wow. Yeah, you know, you're making me think that, like tying a lot of this stuff together, I mean, you know, we work at a software company that, like yourself, does a lot of consulting and working with people, but also sells a lot of products.

So we get, we spend a lot of time paying attention to the products that we want to get people to buy from us and to use and make their life better and improve the world. But there's almost, if you were to think of a, how do we, if going back to my thing, like, if most all the executives or managers and all the way up to probably like shareholders of companies are like, software is really important, and that's how we're going to survive and thrive. And, and you kind of work down from that, probably there's a, maturity model is the wrong thing, but there's probably a path of questions to go through. And like the first one is like, when I fill out a ticket to like get a, get, get a machine, do I have to say the expected load on it? Right. And, and like, even, even before that, I mean, there, there, there could even be like, just looking at the full, like, you know, we're always emphasizing. If you want to improve your software, you have to look at like everything from like idea to a user clicking on the software in production, your, your path to production. And yeah, whether it's gold or platinum or whatever metallurgical color, the path is today and part, maybe there's even a, a, a, a bigger picture, which is like, you've just hired a developer, right? And one of the questions could be, so when you ship them a laptop, is it a developer laptop or a, we don't trust you employee laptop and like spend the first week fixing that. And then we'll come back and just like, kind of like a series of things where kind of like the, the, the, the, um, you know, the, the position that you were saying where like us in the tech world are used to getting a developer laptop. Although when I started at Dell, I did have to file a ticket to get the, you can actually do things, but that's a hardware company, uh, which kind of makes a little more sense or not. Uh, but like, it does seem like there's this series of things that once you have management in a company wanting to have developers, I'm trying not to call it like remedial to be kind of pejorative about it, but it's just like, yeah, this is a brand new thing that you're doing.

Like, and here's the way it goes through the same way that I would imagine, hopefully this doesn't have to happen too much anymore. And, but I would imagine in like, you know, the late eighties and nineties people, car manufacturers are probably like, yeah, we'd love to have high quality and sell lots of cars. And I read that lean book and I don't know what's going on. Right. Like, and so it's sort of like, well, let's go through all the steps, right? First, you got to put socks on, then you put shoes on, like, and all, all the things until you get to the actual practices of it, uh, which probably is helpful to do for people. Organizations who treat their developers like a cost center instead of like a, um, valuable cog in, in the drive train. Yeah. I mean, maybe crazy idea. Maybe one of the things is to, is to put some people with actual development experience in, in that C-suite and I would say, look, this is why you're making these guys miserable. Yeah. They may not, they may not fully understand the, the ramifications of it. And because at the end of the day, like you can spend a lot of time trying to improve your corporate culture, but you know, if you do give developers, uh, limited permissions and resources and, and make them feel comfortable. And if you feel too restricted, then that is, that it's another way of establishing culture, right? You're there's a statement in that. Yeah. I don't trust you. I don't trust you. It's the statement. Yeah. You know, that's, that's an interesting thought. And be, it'd be interesting to hear what, what, what y'all both think about this is, is, you know, as at least, at least when I was kind of like taught as a developer, one of the principles is to have a very strong separation, like causal separation. Between like tools and process and culture, right? Like, you don't, you don't want your tools driving, forcing you to do something or limiting you. However, it does seem like your tools determine your process and culture and kind of vice versa. Right.

Like, I don't know, like, I don't quite have the relationship figured out between like, how important tools actually are for the type of process or culture that you want to do. And, and in the other direction, like it's, it gets a little confusing because people are very obsessed with like phrases like, you know, technology isn't hard. People are hard, but it seems like they're both very hard. And then you combine them together and they're impossible. Yeah, I mean, you mentioned that also somewhere in your article, you spoke about, you know, having a good developer portal, it doesn't mean that you just add like another thing alongside, I think you mentioned like Confluence and SharePoint, like you already have all these things. It's not a question of like slabbing on another one, like you really have to consolidate. Yeah, because again, everything in that article I've seen somewhere, and you'll notice I didn't name any guilty parties, but everybody's guilty. Yeah, everybody's guilty, but you know, they kind of, there's this kind of realization, hey, people can't find the need and the thing is, we'll do a developer portal. So are you going to get rid of the other stuff? No. Well, no, it's all you've, now I've got three places to look in and I still can't find what I want. You know, it's. It always astounds me that, you know, you can go to the internet and go to, you know, any, choose your favorite search engine and type something in and the results are, it's so easy to work with the results. And yet you go into an internal corporate search engine or search site and it's, you know, it's like, what is, what's going on here? I don't understand this. Weirdly, I mean, someone explained this to me once. That actually enterprise search is much harder problem than internet search. Because PageRank, you know, the famous Google algorithm works at internet scale. It doesn't really work at corporate scale because you haven't got enough kind of people doing things and linking to other things and all the kind of signals.

How interesting. That you can use to build your, your rank index that you have on the internet. And that's why, yeah, corporate searches is a disaster all the time. And it has been for as long as I, as for as long as it's existed that I've seen it. It's, it's never worked. I've never seen one that works well. There's probably also, there's probably also a lot of that, that lack of trust of just, you know, like governance and security and like the idea of, like, I remember, I remember in the, back when I was a programmer, like the, that Google mini search, mini search appliance came out. And I remember we got one and plugged it in and there were several people who were like, oh my God, I didn't realize people could find that. Like, you know, and so there is, there is a certain amount of governance that probably nowadays, thanks to like SharePoint and other things, people have gone through a decade or so of worrying about it and thinking about it. But it is, uh, that, that, that, that, that is an issue, but the point you two are raising is, is, is right. It's just like, I mean, it goes back to just the basics of just like, well, does search work? And I think to be honest, you know, maybe people aren't ready to face up to it, but there's, there's only really, if you want search to work in that kind of environment, somebody has to do some curation to make it work. And I mean, maybe, maybe that's something that AI might be useful for. Yeah, hopefully. Yeah. But you know, at the moment, search just doesn't seem to work in those environments. And, and the only way I can see to make it work is that somebody actually tells a search engine, hey, this one is important and this one isn't so important. Yeah. I mean, I, yeah, I see what you're saying in terms of like the ranking. I, I can see that and I can see how it's kind of like a catch 22, right? Like if you don't have, if you, if you can't, if it can't be trained based on like just people searching and linking, then it's hard to have the same end result. But I also feel like a lot of these search engines don't, they don't even have a lot of hits to begin with.

Like there's other stuff there that they don't seem to find. Maybe it's, I don't know that it's the same technical explanation for why those things don't show up. I don't know. I mean, my, my experience generally is usually the opposite that I can see a million results and not one of them that's coming near the top is the one I want. The one I want might be there, but it's so far down. I'm never, ever going to find it. So it's more of a real. Yeah. Sometimes you're right. There's stuff that you search for and you know, it must be there. Nothing. That's fast. I never thought about it that way. That is a fascinating way to think about it. But yeah, it's true. Like AI, AI can probably help us. Yeah. I mean, certainly that's my experience with the, with, with searching on source. But you know, VMware is that I will get a million hits, but unless I'm very lucky, it's not the one I was looking for. Right. Right. Right. Speaking of AI, are you seeing that yet in your work? Not yet. No, I'm sure it will come. I'm, I'm, I'm not sure what I think about AI, to be honest. I'm kind of still very skeptical and sitting on the fence a bit because it feels like. There's been incredible progress over the last couple of years, obviously. And I'm astonished at how well something like chat GPT understands and parses my sentence and comes back with something. But it feels like they're kind of, they're hitting now the hard problems that are still really hard to solve when the machine doesn't really still understand anything. And you kind of wonder, well, is it, is there going to be a way to get past that and make this stuff genuine, genuinely useful and not hallucinate and not make things up and all the other problems it's prone to right now and also be, you know, eminently hackable because you can't defend against prompt injection the way you can against SQL injection. So are those problems going to be solvable or is it going to be another kind of blind alley like all the others with AI that we've seen?

Because for as long as I can remember, AI and nuclear fusion have been about two decades away. Yeah. It's, there's no, yeah, there's no real system of checks and balances, right? I mean, even today on the internet, like the best you can do these fact checkers, I assume it's mostly manual work, right? Yeah, so there's, I guess we're lacking a, we have to somehow automate a system of checks and balances if that's possible. But it startled me how easy it was. Well, I mean, I've been told chat GPT-4 is better, but I had a conversation a few months ago with chat GPT-3.5 and I was talking to it about Bing because Microsoft were experimenting with putting it into Bing. And apparently there'd been some, the internal code name for it was Sydney, for the Bing bot or whatever they were going to call it. And there'd been problems with it kind of really having quite deranged conversations with people who got widely reported. And I thought, I talked to chat GPT about that. And I said, do you know about the problems with Sydney? And it probably, probably made up this whole story about the development scene in Sydney, Australia, getting messages from people, getting abusive messages. And it was kind of like, you made that up. You literally made that up. Yeah, I think we're still a ways away, I guess. Well, the last little cluster of things from your article, I mean, you know, in addition to the rest of the world that I was curious to ask about, especially with, as you're saying, the kind of perspective of going to different companies or organizations in, often in an early stage of wanting to do how they do software or improve it is like, what, what is the state of like having a, like a, like a pipeline in place? You know, not only, you know, it's easy to say CICD, but like when I've looked at surveys over the years, like a lot of people have continuous integration, like maybe 50% and very few have the delivery or deployment where you can do it at will.

And so, like, I mean, what's, what's your general sense of like where everyone is with that? Because it seems like, this seems like another one of those fundamental things that like, if you don't even have continuous integration, start with that, right? Like, like get, get your build pipeline fixed up. So, I mean, mostly, most of the, the, the projects I've worked on, they, they, they have had, they have had CI. At least some, so having said that we, when we wanted to set, set one up for a modernization project, it was really hard because we couldn't find the resource. And literally, we couldn't find the machine to run it on, you know. Right. Kind of endless problems. And we were told, oh, there's this big, super centralized project that is providing pipelines for everyone. So, you go and look at that and it's kind of, yeah, but we're really overloaded. We can't help you right now. So, it's kind of, there's obviously a belief that this stuff is important, but again, companies, again, big enterprises, like kind of, if this stuff is important, we must have a centralized group who will manage it for everyone. And then you're back in tickets and silos and chucking stuff over the wall again. Wouldn't it be better if you had a smaller group of people who just come and help everyone set up their pipeline in the best way and let them look after it themselves? Yeah, it seems, I mean, there's definitely this whole idea of like running a platform as a product that, and sort of like, especially the angle of like, you should build, so this platform that would automate this pipeline for you, right? Like, you should build it, part of running it as a product means that you're doing it a bit more organically because you have to, you want it to be optional, right? Like, it's not mandated that everybody must use it. And so, therefore, automatically, you have to build a product that people will want to use.

And that implies that you have a feedback loop where you can show it to people, have people who recognize that it's the better alternative of the ones available, begin to use it, give you feedback, and then you can begin to improve your own, the platform that you're managing in this way. So, it has to, almost by definition, if you're going to run your platform as a product, it sort of can't be the central silo, kind of waterfall approach. It has to be an organic one where they are talking to developers. Yeah, and that's interesting because, again, I'm on an engagement at the moment with another bank. I seem to get all the banks. And they've actually built, or someone in the organization has actually built what looks like a pretty nice platform internally. But they don't really want to talk to the developers. They're just saying, well, here's the platform. So, what should we, how should we use it? It's your application. You decide. These are the features. And, yeah, that's great. But you kind of think, well, actually, even Amazon don't do that. If you go to Amazon and say, hey, I want to run my application on your platform, they're going to help you. They're not going to say, here's our platform, go get on with it. They're going to try, you know, depending on how much money you've got to spend, they're either going to point you at some resources or, you know, if you think you're going to be a big customer, they're going to send someone on to speak to you and help you draw the diagrams. Right, which kind of, like, speaks to probably the way that companies define the incentives and the metrics around that platform team. They're somehow not structuring that incentivized. You know, Amazon is probably the first platform as a product. That's literally what it was. And, you know, I don't think everything they do is fabulous, but I think if you don't kind of look at some of the lessons they've learned the hard way, you're making a mistake. Yeah, you know, I never actually thought about it that way. I wouldn't have thought to put it in those words, but you're right, right?

That is, because that's, after all, that is what public cloud was. It was literally platform. Literally the first one. And we're going to put this stuff up and we're going to get people to pay for it. Yeah. And hopefully they, yes. And although I guess the road from giving you the car pieces so that you can build your own car to actually, like, the DevX, it is a platform as a product, but it's not a DevX-focused one. It's not, I mean, it's, and it's kind of interesting because I started working with AWS, I guess, in about 2011, 2012. So, not right at the beginning, but fairly early on. And it was interesting watching how fast it matured and how kind of Azure, Microsoft started at the other end. Because they did start with kind of, we want a developer experience and you will build your applications in this way and they will run brilliant on our platform. And they were forced to gradually head down the stack and start offering more and more infrastructure as a service. And Amazon kind of went in the wrong, opposite direction. They started off with the infrastructure as a service, but realized actually people were going to want a platform. Well, the other thing about these companies offering their infrastructure as, it's a form of platform as a product, because the other challenge that they have is that they have so many different customers. Whereas, if you're in an organization, your use case is much more specific. And so, there's also this idea of platform, this whole idea of golden path, right? That has become sort of the new phrase with Spotify and Backstage, where you want to provide a default, a sensible default. Like, you know, some options, but it should be more obvious. Like, what is the one that this organization wants you to follow or something more opinionated and simplified? Yeah, make it easy. Make it easy. Again, the problem with a lot of these, you know, getting the kind of the, if you like, more mature organizations that I'm working with,

their software stack goes back probably quite often to the 1980s and everything since then. You know, again, if you look at, if you look at a bank, a typical retail bank, certainly in the UK, they built their, what is now their core banking system. The core of that, they probably built back in the 1980s on an IBM mainframe using COBOL. And that's still running. That's still at the heart of the business. And everything else kind of has to be fitted around that. It does, it does mean they have peculiar challenges of their own that, you know, companies like Spotify just don't have. Also, Spotify goes down for a couple of days. People are going to be upset. But when a bank goes down for a couple of days, and we've, we've had that in this country, we've had, I know NatWest had a big outage about 10, 15 years ago. And there was another of the retail banks, I think it was HSBC or Lloyd's also had big outages that went on for two or three days. Because people are not just inconvenienced, they're furious, and they can't get their money. Right, right. And it has, yeah. So banks, from that point of view, I understand why they tend to be a little more conservative in their approach. Correct. Yeah, but they're, yeah, yeah. I think the ask is, from a dev experience perspective, it's to differentiate sort of an experimentation area from production. Yes, absolutely. Give us a play pit, but we treat production seriously. Exactly. Well, great. Well, that, that was, that was, well, I, I, I, I came up with a bunch of questions, which is always good. Mysteries to solve. Like, like, for example, like, we kind of hit on the other mystery, which I often forget is a mystery out there, which is, you know, you're talking about, like, Azure starting out as a platform as a service. And then Google was the same thing, right? Like, I mean, famously, I think the people who built Kubernetes had built App Engine or something like that.

And, and then, of course, we have the old Heroku and then, you know, recently, like all the passes and the, the pre, kind of pre Docker, pre Kubernetes era. And there is like, there's this constant thing of like, I don't know if developers actually want a platform because we keep giving it to them. And just like, just like kids who, you know, say they want some food for dinner and then they don't, they keep sending it back. And I don't, I don't know what to give them. I mean, I, I, so I, my first experience with, with Cloud Foundry was when I joined Pivotal. And I kind of, after a kind of initial, well, why would you want this? Once you actually touch it and use it, you think, actually, I don't ever want to do anything else other than this ever again, frankly. It's the, it's the thing of like, once you can get the kid to try the food. Yeah, exactly. Maybe they'll eat it. This is back to my, my, my, my, my theory that, that, you know, the, this, this IT industry, the tech industry is like the fashion industry. Kubernetes is terribly fashionable and sexy and everybody wants it. Even though actually, I mean, frankly, it's a pain to use. I think. I've been forced to move from Cloud Foundry to Kubernetes. I have not found that enjoyable. Yeah. Yeah. It's very interesting. It's lots of fun and I can build interesting things, but it's like, if I could not do it, I'd be happy not to do it. And, and, you know, going all the way back to the beginning, right before the end here, you know, that's, that's part of what drives the, why did we need more than COBOL question. It's sort of like, it seemed fine. And now here we are, but I mean, and obviously, you know, the, the, the, it's not only a rhetorical question, this, whatever this, how can we can never, not never, that shouldn't be hyperbolic, but how come it's so difficult to get a platform into the industry? But it also, the, the positive side of that is, as you were alluding to, is there are these like incremental improvements in how things are done when you move technologies over.

And, uh, it's always, I, I don't know. I don't, I don't think in, in software development, we have a good understanding of, I don't even have phrases for it, but like, it's not like creative destruction. It's creative, some other word of like, how do you, how do you evolve something and moving it, move it forward without it being like, let's just knock it all down and build up from the beginning. Or, or maybe it does happen, but it is a, uh, a strange evolution of, of things. But anyhow, speaking of strange evolutions, this conversation has evolved in exciting, fun ways. And, uh, if, if someone wanted to, uh, you know, we'll put a link to the, uh, the article we've been talking about and, uh, links to other things. But, uh, is there a place that you would point people to if they're interested in hearing about more, whether it was about your illustrious career in Cobalt land, what you do in, in consulting, uh, you know, nowadays. You can find me on LinkedIn, but that's a bit bare, to be honest. Um, although that does, that my LinkedIn profile does at least now have links to, to most of the things I've written over the last few years. Well, that's good. And, and we'll put in, uh, we'll put in the links of things we know about because it's, uh, it's good stuff. Like it's, it's a good span of things to, to, to, to, to, if you want to Google me, you will not find me as Paul Kelly. Paul Kelly is an ungoogleable name because there are so many Paul Kellys and they're nearly all more famous than I am. Which is the most important, Paul? We have to, we have to tell, we have to let Google know that you are the most important. Well, if you, if you add the word Cobalt to the search term, Paul Kelly, there's a good chance you'll hit something to do with me. Exactly. On the first page or so. Or I guess VMware at this point. And possibly VMware as well, yeah. Yeah. If you do, uh, yeah, we will link it, but if you do Google Paul Kelly VMware, you should, uh, get the hit to his article published in April, actually a developer perspective on developer experience, which touches on all these ideas of what exactly is developer experience.

What does it mean to the business? What is the tension with security? What is, uh, running your platform as a product? A lot of really great material covered in this, uh, in this read. So I would definitely recommend that folks go through that. Well, great. Well, thanks again. And, and, uh, for, for people listening, if you want to find all of that, you just go to, uh, tonsutalk.com and, uh, you can find the show notes for this episode, or if you want to watch it in YouTube or whatever, you can look it up there, but it's all over the place. So, uh, with that, we'll see everyone next time. Bye. Yeah. Oh, thank you so much. Thanks for sending. Thank you for having me. It's been, uh, it's been a pleasure. Our pleasure. Yeah, it was great.

Tanzu Talk · 🍕 Get more pizza.