The Twelve-Factor App, with Nate Schutta (Ep. 98)

Episode 83 · 1:12:48 · SRT · archive.org · all Tanzu Talk

So, here we are, as I was joking earlier, live to tape in Paris. I don't know what part of Paris we're in. I don't know. A fancy part? You have some sort of undead sculptures guarding your hotel room. I do, I do. It keeps the spirits away. How would you describe those? You don't want to see them in the middle of the night, I guess. You know, they would haunt your dreams. Uh-huh. You know, maybe they'd be parts of your nightmare. I'm not sure. They look like they're unraveling. Well, one of the things that I realized on whatever date we got here, the placemat changes out front. Oh, you mean the mat? Yes, yes, the mat. Where the undead eat? Where the undead eat. It tells you whether it's raining or not because, like, one mat will say rain and one mat doesn't. So, apparently, that's what you use for checking the weather is you just walk out of your hotel and if it says rain, then you should probably go grab your umbrella. Right. It's one of those things where you're shaving off seconds to optimize. Exactly. Right before you hit the actual outside. Right, right. There's your placemat. But, see, then, as a software guy, my thought goes to, well, how do they decide when to put out the rain mat and when not to? You know, I mean, what's the percentage? You know, what does the weather guy have to tell them? Right. To where they're comfortable. You have to put out the rain mat versus snow. They could have an internet of mats. Maybe. It's on the internet. Wouldn't it be nice if they just swap them out? Yes. And I don't know how frequently they swap them out either. Well, you should ask. I'm sure they'd go to the concierge. Believe me, we could go dial at zero or whatever and they'd answer all sorts of questions. I have a lot of questions about your match. And the undead that guard the hotel entrance. Exactly. Did you notice the Apple Store is literally, like, right next to us here? No. We're in the same building as the Apple Store. I'll do some Apple Store tourism. We'll have to go to the Apple Store. Yeah. Well, why don't you introduce yourself? That'd be a good idea. Hi, my name is Nate Schuder. I'm an architect, advocate kind of guy here at Pivotal and do all sorts of random things in the cloud space and architecture space. So we've been in several sessions, including today, where you talked about, like, 12-factor development. I did. And things like that. And I've listened to your talk a few times and I still have questions. Good. Not because of your lack of information. Hang on. Wait. Wait. Maybe that's not good. I'm not sure if I was there or not. But, you know, I think more specifically, like, listening to your talk made me realize, like, I always have a lot of questions about 12-factor stuff.

And so, like, I don't know. I'll give you my version of the history. And I need to go look this up because I seem to remember when I was an analyst talking with Heroku at some point. There's two things I remember people went over with me that sounded like just a bunch of nonsense at first. One of them is I remember at some point someone was talking about, like, reactive programming or something. Sure. And it wasn't reactive. What was, like, the thing before reactive with event-driven stuff and event sourcing and CQRS? Something like that. I don't even remember what it was. But it just sounded like a whole bunch of, like, that's a solution for a problem that, like, five people have. Right. And then the other one, I think at some point someone from Heroku went over 12-factor stuff with me. And this was a long time ago. And at the time, it seemed sort of, like, not applicable to anything except Heroku. And so it's, from that perspective, it's kind of amazing that it's, well, this is sort of the first thing. It's, like, evolved to be, like, here's a prescription of if you are, I guess, like, doing containerized applications. Although I don't think it ever says containers. No. But, of course, that's what Heroku was. Sure. So that's kind of how it came about. Yeah. But it's, like, if you're doing things, basically, the way I've sorted it in my mind is, like, these are, like, 12 thing comments we have. If you're doing, you're deploying your application in immutable containers. And you basically want to do stateless things. And you also want to do continuous integration and delivery. And then there's some, and then there's another cluster of stuff that kind of falls out, which is, I think they say, externalized configuration. But it's sort of, like, instead of manually configuring these things, what you should do is, like, make it so the configuration is injected to them. Right. And, you know, I think conveniently when this came out, this just coincidentally happened to be that you had to write applications for Heroku. Right. Right. But it's evolved far since then. And so, like, I mean, how do you think about how you kind of chunk out these 12-factor things? Because I don't feel, I mean, if you go to, like, I'll just keep talking once I ask a question here.

But, like, I feel like if you go to the 12-factor site, you read it, and you're still kind of left wondering, like, but why? Sure. Right? Like, why would I do all of this? And I think the, I haven't read it in complete in a while, but I think the only thing that they say is, like, well, then it can scale. Well, so you can make an argument that a lot of the things that are in the 12-factor are just good design principles. Yeah, yeah. Like there's loose coupling. Yes, yeah. I mean, this has been something that's been kind of, I guess, coming into form in my brain. But the longer I do this, the more I believe that, you know, I don't quite know the best way to phrase it. So I'm just going to call it, like, the zeroth law of software engineering, which is, like, high cohesion and low coupling. You know, so many of the things we're trying to do are just fundamentally, how do we get high cohesion and low coupling? I mean, I would argue that microservices are just high cohesion and low coupling brought to SOA. You know, so how do we do SOA in a high cohesion, low coupling kind of way? You know, so a lot of the stuff in the 12 factors, you could say, is just some fundamentally good design decisions. You know, yeah, you should have your code in a version control system, probably Git. And, you know, you shouldn't be copying code from one place to another as a way of sharing it. You should export it into a library. I mean, these are just good ideas in general terms. So you should have modularization. Imagine that, right? Yeah, yeah. Who could have predicted that? You know, I look at the legacy apps that I've tried to move to the cloud. And, you know, as we talked about today, I mean, that you're going to violate some of those 12 factors because we built these applications before the cloud existed and before we knew that's how we should design applications. Right, right. And so, like, what would be examples of those? Well, so you take these applications that we wrote five, ten years ago, and one of the violations I see all the time is long startup because we would go out to our database and we would cache a whole bunch of stuff into memory. You know, we'd go get states and countries and provinces and, you know, business codes and all this other stuff that we knew wasn't really going to change very often. So we did load all that into memory instead of using a distributed cache. And that was fine when all we did was reboot our apps, you know, maybe nightly.

Yeah, yeah. Maybe weekly. Big deal. Well, you then go to a cloud environment where these things are going to be cattle and they're going to be disposable and an instance is going to go away and come back because maybe it got sick or maybe I need ten more instances to handle this big rush of traffic I got. And that three, four, five, ten minute startup time causes me issues. I can't scale that up. I can't bring those new instances online because if that health check hasn't completed, the router says, oh, well, you must still be sick. I'm going to spin up another instance, another instance, another instance. And we kind of lose that whole value. I guess maybe that's the – to cut you off to the thing I'm always fishing for with the 12 factors is like it doesn't – it's almost like you have to reverse engineer what the operating model is from it, right? Whereas, as you were describing, everything you said, at least in my mind, comes from an operating model of I am going to be running containers just willy-nilly, I being a platform or the run – I don't know what else to call it, the run space. The run time. And, like, my job as the run time is going to be, like, I might kill it or I might bring it up. And the way that I scale it, like, I always think it's a trick of scaling. The way that I scale it is state is someone else's problem. Sure. And then I just have these, like, tiny stateless things that I can just in parallel scale up. And hopefully the state people over there have figured out what to do. Yeah. And so, like – but, you know, I feel like over the years – well, not over the years, but the 12-factor stuff never begins with – and necessarily so because it would be really boring. But just to say, like, here is the run time paradigm that you're operating under because so many of the things in it, as you were kind of going over before I cut you off, sort of makes sense, right? Like, if I'm going to be bringing containers up and down, I can't have a two-minute startup time. And so, therefore, because I have this type of run time environment, I need you to do your applications in this way. And I guess the – it's not like a beef, but the cognitive, like, grading I always have in my head is – I feel like it's easy to think of the 12-factors things as what you say, causative.

Like, if you write your applications in this way, good things will happen. Whereas, like, you've got to bring the good thing run time environment first. Well, and reflect on where the 12-factors came from. Yeah, yeah. The characteristics that were shared by successful apps that were running on a broker. Exactly. You know, so it's a little bit of, I guess, maybe a cart and a horse kind of a problem, right? You know, so it's like if you're going to design an app for this platform, here are some of the things that you should take – Right, right, right. You know, if you're not going to design for that platform, well, then you could ignore some of this. Again, I would say a lot of it is just good design principles. Yeah, yeah, yeah. You know, I mean, maybe you don't have to worry about the long startup time if you're not in one of these kind of cloudy environments. But I'd be curious to know why you're not. Yeah, yeah. Yeah, I was – I mean, you're right. Like, I was separating them out into, like, three buckets. But there's an initial bucket of just, like, how you should write good software. Right, right. And then it seems like the next bucket is sort of, like, other ways you should write software, right? Like, like, one of them is, like, you know, you should have one process per – you know, you should focus on a small process instead of, like, one giant multi-process thing. Or, like, as I think we're supposed to say, the monolith, right? Or whatever. And then there's, like, there's, like, the other group that are, like, what would you call it? Like, deploying, like, you should do continuous integration and, like, you should – you know, things like that. And then the final cluster is pretty small. It's basically, like, logs. It's like, sure, we all agree on logs. We got to do logging. And then I guess there's, like – How we handle that in the cloud. And then this last one is, like, a – it sort of makes sense, but it's an oddly specific one, which is – what is it? Like, your admin thing should be a process. Yeah, yeah. Like, it's just sort of, like – That one always kind of struck me as, like, they just – they're, like, we're at 11. We can't have 11. We can't have the 11 factors. So let's throw another one on there just for fun. Yeah, which, I mean, that's another one where, like, I assume what that means. I mean, I should have the thing in front of me. It's, like, it's almost – instead of, like, logging into a box to run shell scripts, you know, not that a box exists, you should – these scripts should be their own containers,

these jobs that execute to go query the system and look at things, which, again, is, like, sounds cool, but that's pretty specific. Right. Yeah, I agree with you. That one is oddly specific. You know, I don't know the history on why that one popped up. Yeah, yeah, yeah. You know, so it is a little bit of a head-scratcher. You know, but, again, I mean, I think where we run into some interesting challenges is if I've got this legacy application, which will violate some number of these factors, you know, how many do we need to fix to move into the cloud? And, you know, I've had some interesting debates with folks over that. Right. I had one architect tell me that, well, you know, if an application doesn't need all 12 factors, we shouldn't even bother trying to put it in the cloud. And, like, well, that's pretty extreme, you know, because the odds of us taking a legacy application and essentially rewriting it to be 12 factors compliant. I mean, that really isn't a good business return on something like that. And so it becomes more that pragmatic, well, how far on that continuum do we need to get? Yeah, yeah, yeah. And what gives us the right mix of business value to effort to say that that's the logical ending point? Yeah, yeah. I mean, tell me if you found this to be the case. But it seems like you could equally kind of cluster them out about, like, these are the most important ones. Oh, absolutely. So do these three. Yeah. And then it's also important to do these other ones. And then a further division is, like, when I look at them, it seems like there's sort of, like, process-oriented ones. And then there's, like, very technical things, right? Like, your loose coupling is very, well, technical-oriented, right? Versus, like, what's one of the process-oriented ones? Like, I mean, even the idea of, like, having trunk as where you check something in. Like, it's a tool thing, but it's also a very process-oriented stuff. And so I would imagine when you're thinking about legacy stuff, to some extent, the technical things are more important. Right. Because, like, if you don't get your legacy chunk set up technically correct, it, like, just won't run. Yeah, you're absolutely right. I mean, well, like, the long startup time is a perfect example. Yeah, yeah, yeah. If you've got a five-minute startup time and we can't get that down to a few seconds, 10 seconds, 15 seconds, whatever is reasonable within, you know, how it's configured.

Yeah. You really lose that ability to self-heal and auto-scale, and that kind of diminishes a huge advantage of the cloud, at least from my point of view. Yeah. You know, so better get that one right. You better switch the logging over or else you're not going to have any logs because that instance you're writing out to that file system, it might get blown away later today. Yeah. So you're not going to be able to go look at your logs because they're gone. Now, the interesting thing to me is a lot of these things actually end up being better for us as developers because, like, I get this big stream of logs and I get way more powerful tools to actually go interact with them. And it's like, hey, this is a lot better than me trying to use a text editor and do a little, you know, command F to go try to find some random string and 10,000 lines of log output. Yeah, yeah. How much time have you spent diagnosing stuff with logs just, you know, in your illustrious career? You know, it's probably more than a year, I would think, honestly, of my whole life. Yeah, just total clock time. I'm sure because it just – you spend so much time staring at logs trying to figure out what happened and what went wrong. And the funny thing to me about logging is it's that hard thing to find the Goldilocks point. You know, you either underlog and there's not enough data or you typically overlog and there's just so much noise you can't find the signal and you can't find the thing that's really meaningful. And then you start doing really crazy things in your logs to try to make stuff stand out and then you get these weird anti-patterns where there's, well, if you put five asterisks in them, you know, or whatever, and people start looking for weird patterns. And, yeah, it makes me shudder a little bit. I have never thought about, like, the cumulative effort I've spent on logging, but it would not surprise me if it's more than a person. Yeah, well, I mean, I used to work on systems management software and then when I was an analyst, cover it. And it can be a really annoying field if you, like, study it for more than 10 years because the same sort of things – it's not that the same exact things repeat each other. It's that the tools you have, they're almost, like, in a line and then one tool gets really good and you kind of forget about it.

And then you move on to doing the next tool and then eventually you forget about it. And each successive way – like, it's – I think basically what the ordering of the tools is always, like, like you were saying, you know, like, you go on to the – you get to the machine and you, like, pick up the metaphoric objects you're looking at and you look at them. Right. And you poke and prod them and, like, you know, you go look at this configuration file and go see what the process looks like and this stuff. And then you can kind of automate that where you're collecting it. Yep. And then at some point after that is, like, you got to have logs. Right. And you get – and then, of course, this part of logs is, like, now that I have all these logs, I need to make sense of them. Right. Not only logs from one unit of suffering, but, like, you know, I've got a cluster. I've got to collate them together. And then, like, the most advanced form of that is, like, and then I need to trace one single process through the logs. And so you go through that cycle. And then another characteristic of the systems management cycle is, like, so then the log people say that the pick-it-up-and-diagnostic people are complete idiots. Yeah. Right. You always are blaming the people before you, like, some sort of terrible revolutionary – We're good at finger-pointing, aren't we? Yeah, yeah. And then, of course, after that, you have the – I don't know what to call it, but you always arrive at this point where you're, like, we're going to use complex analytics and just, like, consume all this stuff. Yeah, yeah. And it'll just tell you what's going on. Right. And then, of course, those log people and those monitoring people are idiots. Yeah, yeah. And then, of course, no one ever solves the complex analytics thing. But anyways, I go over that whole little mini rant there to, like, looking at the 12-factor stuff, the idea of, like, having logs, right? There's an interesting, like – it's not that it's a tell. I sound like I'm being all cynical. But it's sort of, like, it's, like, the thing that's available, right? Like – It's the tool we've had. Yeah, yeah. Like, if you have immutable containers or however they call it, right, these little containers that are coming and going, you'll, like, never get a chance to log into them. No. And check them out.

And then, like, you could – Or the one you needed is gone. Yeah, you could – The one that had the problem, well, we blew it away. Exactly. You could sort of monitor them, like, query them to know what their state is. But really, all you're ever going to have is the logs, like, afterwards. So, like, that's what you have to deal with. So, therefore, logs are important. And I guess the point, since it's 12-factor development or apps and not 12-factor sys-admitting, right, like, is that, hey, developers, you should be logging. Right, right. You should log a lot of what you're doing. I don't know. But all of these factors, like, I spend a lot – I spend too much time over the years trying to figure out, like, why that factor? Sure. And how does that map into the platform you have? Which then I'll come up with a question here instead of me just talking. But, like, how pervasive do you think the use of 12-factor stuff is in the types of runtime environments people use nowadays? It's a good question. I mean, I think anybody who is moving an app to the cloud has certainly done that evaluation on the app to see where it fits. I think the number of truly 12-factor app-compliant apps is pretty small. And, like, when you say the cloud there, like, what does that mean in that context? Well, whether that's on-premises, whether that's public cloud, you know, whether that's in a Docker container. I mean, I don't think it's – But just a containerized application. Right. Some kind of environment where you say, here's my code. Please run it for me. You know, do your thing. You know, I mean, I did this at my last job. We evaluated our 400-plus legacy applications to figure out whether they're good or not good candidates for the cloud. And so part of the metric we ran them through was 12-factor compliance. You know, they weren't the only things we looked for. There were a handful of other questions that we were asking, but that was the core of it. And then we used that to basically – we had kind of a scoring algorithm that came out of that. And then we essentially bucketed the application into this is a good candidate, this is a candidate, and this is a bad candidate. And then there was a sliver that fell off because they had some constraint that meant you can't go to the cloud.

And was it just like a spreadsheet with 12 rows and then columns of – More or less. Or whichever way you oriented it. Let me put it this way. The big lesson I learned on that is if you're going to do data collection, you should just build an app to do the data collection. You should have an administrative process in its own process. Absolutely. You should build a little 12-factor app. Because what I discovered very quickly is like it was – the spreadsheet wasn't a bad way to get the data from people. The thing that made it challenging was when you had 400 or 500 or 600, however many we got back, you then have to make sense of it. And what I discovered very quickly is like senior director type people just assume they can ask random queries and then you can somehow just pull that out of the spreadsheet. And so I would literally get these questions like, can you tell me how many apps violate this factor? And if it was in a database, well, sure, select star from where – it's like that would be easy. But it's in an Excel spreadsheet. Yeah, yeah. Now, luckily for me, I had a friend, a guy who sat right next to me who heard me talking about this one day and he's like, well, I know how to manipulate Excel spreadsheets. Sure. And so he did a whole bunch of macro-y stuff for me. You just need a pivot table. He saved me years, frankly. I mean, the stuff he did for me was fantastic and it helped me kind of sort through these things. But the lesson I took out of that is you really want to get that into a database so that you can actually do some searching on it and do more interesting things with it. But the part that I found really fascinating out of that is we had done enough pilots that we kind of had a sense for how long it took to fix any of those 12 factors that you violated. And there was a collection that was pretty common, you know. I mean, we ran into the long startup time, of course, and the hard-coded configuration and stuff like that. And so we knew after doing four or five of these that that was probably about three days or that was probably a week. And so we could actually add that in. And I had that as part of the formula that spit out the other end and said, okay, well, based on the factors that you violate, it should take you about four weeks to lift and load into, you know, for us, it was PCF.

You know, I mean, it was a rough swag. I mean, I was never going to say 100%, yeah, you're going to hit that number. But we at least had an order of magnitude. And so we could use that then to feed into the prioritization part of this to say, well, these apps are actually fairly easy to load. And then you kind of map that to the business value. You get out of the app and, you know, is it going to be with us for the next five years or is this going to be retired in six months? Yeah, yeah. So you could start to make some decisions about how do we prioritize all these apps. Where to expend your effort. Exactly. And that's where a lot of organizations are at. You know, you've got this huge legacy set of apps that are delivering business value for you. You want to try to modernize them, but not all apps are equal. Right, right. Some apps are more equal than others. So let's figure out what those are and then get that into a nice prioritized list and go forth and prosper. So how do you evaluate each of the factors? Well, so again, some of it was pretty easy because like logging, we knew they were all going to violate logging. They were all writing to a log file. So that was an easy one. Yeah, instead of standard out or whatever. The biggest challenge was realizing that like as an architect, I couldn't evaluate your application. I didn't have the subject matter expertise to do it. And so we had to identify for each of these 400 projects who was the subject matter expert. And that person then had to do the evaluation. Where it became challenging is some people were very diligent and they did very good work. And the data we got back was very good. We had other people who copied and pasted and really weren't paying attention and in a handful of instances left a whole bunch of stuff blank. And it's like, well, no, we really need you to go in. Yeah. We really need you to say, do you do this or not? And part of what we had actually done was break the 12 factors out to be a little more verbose. You know, so there were sort of individual bullet points within each factor in some cases. And so that kind of made it a little more nuanced, you know, in terms of how that was done. Yeah, yeah. Well, that sounds useful. It was kind of interesting to see how that was done. Now, we quibbled maybe a little bit about because that was our enterprise architect team had actually built that out.

You know, so I don't necessarily know how they came to some of those conclusions about this is these five bullet points somehow are part of this one factor. But that gave us a little more nuance in terms of, all right, well, do you really violate that or not? And how much of that factor do you really violate? You know, which is why I don't think we had any apps that violated all 12. I mean, I can't honestly think of too many apps that would because you'd have to basically not be using version control, which I really want to believe is no longer controversial. Yeah, and back to what we were saying, there are several of those criteria that are less about the app and more about the process they're using around it. Sure. You know, or even just good design. I mean, some of it's like you're probably doing this almost by accident if you have some design chops, right? Yeah, yeah. So then what does that end up looking like? I mean, since you did so many of them, like, what's the feel you get for applications in aggregate? Like, are there usually, is it like only three of these factors are going to be followed mostly on average? Or like, is it kind of like, for the most part, everything's fine? Or like, where do the problems come up that you have to work on? Well, so you start to discover that there are certain factors that are violated almost universally. And there's other factors that are followed almost universally. Right, right, exactly. Yeah, which is really fascinating, you know. And it's not 100%, obviously, but, you know, we kind of knew there were going to be a few, like the logging and the long startup, you know, that most of our apps are going to violate it. Dependency management was another one that was very frequently seen because, you know, we didn't have to worry about it. You're just like, well, it works. Don't worry. It's probably on the class path. It's fine. Right, right. Nope. Now you need to tell me if you need that jar, that DLL, whatever it happens to be. We have to explicitly define that as part of our app. You need to manifest for it. Exactly. You've got to be a lot more precise. You know, so you start to see some of those patterns. And then, again, you can start to develop those rule of thumbs that say, well, it takes us about a week to fix that problem. It takes us about two weeks to fix that problem. And so now you can start to actually get some rough estimates. And that turned out to be really useful, you know, because you had some people who were like, oh, God, you know, it'll take forever to move this app to the cloud.

And then you look at it, you're like, it's probably six weeks. You know, I think we can probably do it. Right, right. And so it really helped people kind of conceptualize that this wasn't a multi-year process, you know, that this was something that could be done reasonably quickly. And then now, okay, well, let's just prioritize it. And does that make sense within the overall life cycle of this application and where the business is going with that? And are we willing to make that investment or not? Yeah, yeah. If we're not, that's fine. But at least we're making it from at least somewhat of an educated place as opposed to just throwing darts at the wall and hoping it works out. Right, right. Or picking cool sounding projects. Sure. Or just putting a whole bunch of names in a bucket and pull it out and say, you know, may your odds forever be in your favor. So what was the original motivation to do all of that? Well, so a lot of it was just trying to figure out, like, as we're going to modernize this portfolio, what does that look like? And knowing that we had such a big portfolio, we needed to do this in a more methodical way. Yeah. You know, it couldn't just be let people randomly do whatever they wanted. We certainly, as a business driver, as a monetary driver, we're trying to get as many workloads there as possible because it was far cheaper environment for us to run in. You know, we had a situation where being able to shut down regions was going to save us significant money. And so that becomes a pretty powerful driver. So for us, it was really just trying to get our heads around, well, how much of our portfolio is it even reasonable to put in the cloud? You know, so that we could then have those conversations with our senior leadership. You know, so if they were getting this pressure to move all the apps, it's like, well, 10% can't be moved. Okay, so just understand that these guys have a dependency. Maybe they talk to the mainframe in some way we can't abstract away or they've got a third-party vendor thing that doesn't have a cloud option. You just, you can't move this. So it's sort of like an infrastructure optimization motivation. Yeah, absolutely. We're using lots of air quotes here, but we want to move to the cloud. Yes. Because it's going to allow us to spend less on infrastructure and hopefully also work at least as well, if not better.

And so then you go through that exercise of like, well, what are the things we can migrate and things we can't migrate? Now, that makes a lot of sense. Well, and then again, it was feeding that prioritization, you know, because it's like to understand, you know, there's this kind of Goldilocks, right? Where it's like, if this application is easy to move and we get a lot of business value for moving it, put that at the head of the list. Yeah, yeah. If this application is really hard to move and we get very little business value out of it, put it at the bottom of the list. Yeah. You know, there's just no way you can move 400 apps in a month, in a year. So you have to have some sense of what that looks like. And that fundamentally fed into a roadmap, you know? Yeah. So over the course of the next, I think we did it over like three years, when do we want to slot these applications in and what's reasonable? Yeah, yeah. So you work with the architects, you work with the business partners to make sure that they're all on board with, okay, we're targeting your app on the cloud for this quarter of this year. Is everybody okay with that? Yeah. And then you kind of feed that back into the loop to say, all right, this is what we're thinking. And, you know, then the senior leadership can say, well, we want to accelerate that. What's it going to take to do that? Or, no, that seems about right. Or how come these apps are way out here? And then they could have some conversations with those teams to say, why can't you move up sooner? Or we think we'd get more value by moving you up sooner. And it's like, okay, well, you guys go have that conversation because that's what they told me. Yeah. So what do you call that stuff that's preventing the applications from being modernized, right? And I ask because it would be easy to call it, like, tech debt or something, like technical debt. But it's not really that because, like, it wasn't debt. It was just, like, that's how the applications were written. You couldn't have written an app 10 years ago. Yeah. With the cloud in mind because it wasn't an option. Yeah. I mean, I guess it's just, like, legacy characteristics or something. Sure. Or, like, how would you describe those? Yeah. I like that phrasing. Because, I mean, to your point, an application that you wrote 10, 15 years ago, by definition, you couldn't have written it with the 12 factors in mind.

Yeah. You know, it's just not possible. So you can't beat yourself up over the fact that an application violates some of these things because they weren't constraints for us before. And they are now. So part of what I saw there was fear. It was people like, ooh, this is new and scary. I don't want to be the first person to try that. I want someone else to be the first person to try it. And some of that's reasonable. You know, it's – I think about this a lot, ironically enough, with food. You know, there's certain food items that you look at and we eat them and we enjoy them. And you think, boy, the first person who ate this had to be really hungry. Right, right, right. You know, or there's – what is it, like a walnut or something where it's, like, if you don't treat it properly, it'll kill you. But if you treat it properly, it doesn't. I mean, there's some nut like that. I don't know. You have to know how to, like, cook it or something. That might be children. Could be children. Yeah, who knows. But you think about, like, all the trial and error that went into figuring out how to properly cook that thing or how to properly prepare that thing so it didn't kill you or make you sick. And it's kind of remarkable, right? You didn't want to be the first one. Here, try this. I think it'll be fine. You know, there's that poisonous fish that you can get if you're really crazy. Yeah, yeah, yeah. The chef has to know how to do it. Again, the first person who tried that – or if you're learning that skill, right, you're learning how to make the fish that way, do you really want to be part of the test group? Right, right. It's like, sure, you can go to, you know, the place where they train you how to cut hair and you can get a really cheap haircut because the person who's cutting your hair has never cut hair before and get what you pay for, you know what I mean? But if you're willing to make that tradeoff, fine, right? But I think about that a lot even with some of the things we try to do in software. You know, we don't – we haven't reached that maturity level yet where we have all these building blocks that we understand how to use. You know, you think about, like, the architect who designed this building we're in today. You know, they're more or less kind of picking up Lego blocks and snapping them together. They know that this beam can support this much weight in this configuration. If you do it like this, it can do that. And they know all the things about load-bearing walls and all that kind of stuff. So they have these really big abstractions that they can work with. And we historically haven't.

You know, I mean, it wasn't that long ago that if you and I wanted to write a program, it's like, okay, well, the first thing we need to do is write an operating system. And then we need to write a compiler. And then we need to write a text editor. And then we can start writing some programs. Right, right. You know, and we don't have to do that anymore, right? We get our own operating system. We don't have to deal with that. We don't have to worry about compilers as much, you know. And so we've moved up the abstraction tree. But we're still at this phase. And the analogy I like to use actually comes from Calvin and Hobbes. I love Calvin and Hobbes. And there's this great comic where they're driving over a bridge and there's this weight limit sign. And Calvin says, hey, dad, how do they know what the weight limit is on the bridge? And he says, well, Calvin, they drive heavier and heavier trucks over the bridge until the bridge falls down. And they rebuild the bridge and they weigh the last truck and then they put the sign on. And he says, oh, okay, that makes sense. That's kind of what we do in software. Yeah. It's like, is this a good use of this technology? I don't know. Let's try it and find out. Yeah, and then hopefully you build a bridge the exact same way the second time. Hopefully, right. And, you know, I mean, there is that analogy, which is, well, if you want to know how long it'll take to build something, you know, build one and throw it away and now you know. Yeah. And that's still kind of what we're doing. There's a lot of trial and error here. You know, so you try something out. And one of the things that I've seen repeatedly in our industry is somebody will create something new and interesting and cool. And it'll solve a problem that they have. And they'll talk about it, blog about it, whatever about it. And a whole bunch of other people get really excited. They're like, well, I need to do that thing too. And then they start applying it in places it makes no sense. Yeah. And then it falls over. And then you get the inevitable backlash of that technology is terrible. No one should do it. I mean, you had a tweet about that very recently that I've actually stolen and I use in some of my talks, you know, which is exactly the hype cycle that, you know, and then go back to step one, you know, hey, look at this. Isn't this cool? You know, no one should ever use this, especially if you're trying to, you know, put up a blog or something, you know. And that's part of what we're dealing with is that in many cases, the only way we know if it doesn't work is by trying it and discovering it doesn't work. Yeah, yeah. And learning, hopefully, from that, you know, and then rinse, repeat until we figure this stuff out.

Yeah, yeah. No, and I guess, I mean, that is something that's interesting about 12-factor stuff is it seems to have become, as you say, if you want to put something in the cloud, you should follow these rules. Right, which is sort of remarkable given that it was like just, you know, a marketing web page. So, you know, most marketing, no matter who writes it, is never really that useful over the long term. So, like, you know, to the point of, like, how you write things well and how you're crafting those. And also, like, it's a good way of reflecting on, I think, migrating applications to it, right? So, like, another, like, the way that 12-factor stuff thinks about an application, right? It seems to be that, like, basically, and it sort of, it says, it uses different words, but it's like, first of all, your application is going to be, like, transient. Like, it's going to come and go, so you can't depend on anything being there. So, it has to be stateless or disposable, as I think they would say. Treat it like a cow. Yeah, yeah, right. And then there's a whole lot of interesting stuff scurrying around with the idea of externalizing your configuration, which is interesting. And then, finally, there's, like, what would you call it? I mean, I'm always too sarcastic about it, but there's sort of, like, the big asterisk of backing service, right? Like, to handle all this stuff for you. But going back to that, like, how would you compare, like, I haven't actually written software in a long time. But, I mean, tell me if this has changed any over the past 10 or so years. Like, it seems like you would write a Java application. And, like, you can kind of see how it's almost the opposite of a 12-factor application where the goal of a Java application is there's, of course, no .exe in Java because it's Java and not Microsoft. But, basically, you're creating, like, a giant .exe file that embeds all the middleware that you need and, like, all of the various layers that you need.

And, like, you know, you just start it up in your app server and it has everything. And, at worst, you'll have, I guess, they had ear files, which are just a way of bundling together a bunch of these other files. Let me channel Josh and say, you know, make JAR not work. That's right. And then, but there was something that, like, I mean, and it's used today in Java applications to be 12-factor compliant. And it's, like, they were always encouraging you to pull your configuration out and put it in, like, property files and then, I guess, XML files and then environment variables and things like that. But there seems to be, like, I don't know. That model seems to, I guess that's not the right way to do a cloud application. Like, so how do you, how would you, if you built a war and an ear and all that stuff, like, how do you pull it apart? We would keep it as a JAR. Again, I do like Josh's quote, you know, make JAR not war. I mean, either way, nowadays, you can execute all of them just the same. It doesn't really matter what the three-letter extension after your zip file is. Right. It's all, yeah, it's just yet another way of encapsulating your code, I guess. But, you know, that's a good question. I mean, again, I would argue that a lot of these things, it's just the design we've been trying to do for a while. I mean, external configuration, that's always been a good idea. You know, hard-coded configuration has never been a wonderful thing for us. I mean, I still distinctly remember early, early on working on apps where, you know, we had some of the, like, passwords were in clear text. If you knew how to log on to the right server, you could see the non-expiring username and password. You know, and it's like, well, I mean, we didn't have a better option at the time. You know, that was the best we could do. And you just kind of had a little bit of that obscurity kind of concept of security, which, you know, worked out okay. Now, we wouldn't want to do that today. You know, but that's, again, to me, that's just part of, like, how do we design good software? You know, not so much whether that's going to run in the cloud or whether that's going to run somewhere else and what that looks like. But what I find exciting about this is just that shift from I take my, you know, if we're talking about Java in particular, I take my Java code from one instance of an app server to another instance of an app server that I call test.

And it's on another physical piece of hardware somewhere else. And, you know, I inevitably run into these issues. Well, it worked on my dev server. Sure. And it doesn't work on my test server. Why the heck not? You know, what's different between these two servers? They're supposed to be identical. You know, and then you spend days, hours, weeks trying to figure out what's different. And that is a really unfun problem to solve. Sure. You know, and so, you know, and I think that's so frustrating is, like, all the time we're spending chasing down those rat holes is time we're not solving business problems. Yeah. You know, our customers don't care about those things. They just want us to get the next card done, get the next story done with whatever they happen to be using for requirements these days. And so being able to kind of lift that abstraction up to, well, no, no, this is a unit. You know, you're essentially saying your app server and everything it needs is all one block. And that's the thing that we're, quote, unquote, moving from one instance to another. You know, and, of course, let's be honest, until we get to prod, it's generally not actually physically changing anything. We're just pointing at a new thing. You know, this URL goes here now instead of here. And, you know, who cares? It might even be on the exact same, you know, rack that it was on before, but so what? Yeah. So to me, I think that's really useful to get out of this mindset of I'm going to log on to server 123XYZ. Who cares? You know, that should be somebody else's responsibility, you know, not me. I don't want to worry about that level of abstraction. I want to just focus on, hey, I have a business problem I need to solve, and this happens to be the right language to solve it in. Now, here, take this thing and please execute it for me so my business person can get their value out of it. Right, right. Yeah, yeah. So, I mean, I guess that is the, it's a continuous story of we've been trying to do things in this good way, but we didn't do it very well last time. Right, and then, you know, if where we are right now is in, right, like, when we get to in plus one, we'll have the same opinion. Yes. That, like, the way we used to do things was that we didn't do it very well. You're absolutely right. And that's an important thing to remember. One of the things that frustrates me about this industry is we do a really bad job of learning from the past.

You know, I have this theory that for most developers, time begins with the first language they learned. Yeah, yeah, yeah. That anything that happened before that is ancient history, nobody knows anything about it. This is where all the ideas started from and came from, and they don't, we typically don't reflect upon the fact that, well, no, that idea came from 35, 40 years ago. That's right. And it was a really good idea then, and it's still a good idea now. Yeah, yeah, yeah. I mean, you look at Fred Brooks, I mean, this guy was making statements about our industry 30 years ago that are true today, and they're going to be true in 30 years, and they're going to be true in 100 years. You know, that, to me, is an important thing. We need to reflect upon that stuff. And that's a good point, because go forward five years, we're going to be doing something different. I don't know what it is. I can't predict the future. Yeah. But it'll be different, and you're absolutely right. We'll reflect back on this, and we'll be talking about our legacy cloud portfolio, you know, and we're going to be arguing about, yeah, we're really stupid. Back then, weren't we? You know, and it's like, no, we weren't. We did the best we could with the tools we had in front of us. Five years from now, we're going to know more. Ten years from now, we're going to know more. Yeah. And so it'll just keep evolving, and we need to appreciate that, you know, and we need to reflect upon, too, that a lot of what we do today is just a renamed variant of something we did before. Yeah. You know, I understand the hype around microservices, and I get that. So microservices are, you could make an argument, it's just SOA done right, you know, and we can argue that SOA is not a heck of a lot different from what we were doing at EJB, which wasn't a heck of a lot different from CORBA, and I'm sure there was something that came before that. Right. And they all have very similar characteristics. Yeah, and I guess the thing you're noodling on there is, I mean, it's helping me realize one of the, you know, one of the confusions I have about 12-factor stuff, right, is it's a mixture of, and it's very coy in how it does it, but it's a mixture of technology prescriptions and general principles. Right. Right, and the issue is always, well, the core issue with it is the technology prescriptions don't really ever tell you what the technology is. Right. They just tell you what the desired effect or constraint of using the technology is, right?

So, like, I forget it, but, like, if it was written nowadays, it would probably be, like, package your software in containers. Sure. Right, and, like, done. Like, use a container to package it and use a YAML file to store the configuration and then use this kind of, like, environment variable. I don't know this stuff, really, but use this kind of environment variable passing in. And then use Kubernetes. Yeah, and then when the container gets packaged, open up the YAML file and jam these things into the environment variable so that when it boots up, you've got your configuration injected. Right. Where it is, like, it's a little more poetic. Yeah. 12-factor stuff, it's more poetic in its approach, right? Whereas it has principles in there that are, like, you should have loose coupling. Right. Right? Like, however it is you want to achieve that. And I think, I forget the timeline, but, I mean, I think microservices as a concept started crystallizing after 12-factor stuff, right? So, it would be easy to kind of say that, like, oh, when they say loose coupling, they mean do microservices. Which is probably, to retcon it, is probably true. Like, what we're saying is to do microservices. But it is, like, it's hard to sort out those things that are principles that you should be following versus, like, technology things. And then you're reminding me of one that I always forget. And I only remembered it when I actually looked at the list later today. I don't even remember you talking about it, but you must have. But you alluded to it, which is, like, you should have dev prod parity. Which is sort of, like, that's an easy one to just sort of, like, I don't know if this is a proper use of it, but just sort of, like, whistle past as you walk past it. Which is, because you were just describing, right? The problem that you have all of the time, if you don't catch it, is it worked in, you know, you got that little girl with the firehouse. Like, you got, it worked over here, and now it doesn't work over there. And this is another one of those things to go on, another long monologue here that I feel like was a, what would you call it? A unintended finding of it being Heroku, which was, like, there is no difference between Heroku dev and Heroku prod.

As far as a consumer, it's, like, there's only the Heroku. Right. Like, you can't, so, I mean, there's a difference in configuration, but it's the same exact, like, setup. You might consider something dev. Yeah, yeah, and you might be routing it differently and things like that, but it's the same exact, I mean, unless they're doing something bonkers, it's the same exact thing that you're running. And I think, it's going to end up on a rack somewhere. Exactly. Like, the Dynamo you're using over here is not an instance, but it was just a different, it's just a different instance of the Dynamo here. And you see that in Pivotal Cloud Foundry and other stuff. And I think, like, that's something I never see, I very rarely see people talk about. And even with Pivotal Cloud Foundry, they'll talk about having multiple foundations and things like that. And it seems like the goal is to have the one cloud, right? And then that will both force and eliminate a lot of issues. It's taking variables out of play. Yeah, yeah, exactly. And what's fascinating to me is I spent a lot of my career developing Java on a Windows desktop that was then deployed to some Linux variant. Exactly. Right. It's like, wow, who would have thunk it that this app server running on Windows isn't identical to that app server running on Linux machines. Yeah, let alone configuring the database and the networking and, like, all the stuff. Yeah. And so anytime we add variables, we're just making our job harder. Yeah. And then so that's, I mean, the way that you're detailing it is, like, that's an instance where you could externalize the configuration all you wanted. Right. But if you have, if you're Windows in development and Linux in production, like, who cares? I mean, obviously it's going to be better, but you're still going to find problems. Exactly. And so, yeah, I mean, I don't know, I'm finding that people don't talk about that parity very much, but that seems like, that seems like one of those things that you sort of, like, you should probably do that first because it's going to get you a huge amount of gain. Kind of back to your spreadsheet example, like, whether we modernize these applications or not, if there was extreme parity between the development environment and the production environment, or it was the same environment, things would probably run better.

Right. But anyhow. Oh, I mean, you asked me, like, how much of my time I've spent staring at log files. It's probably equally as much of how much of my time I've spent dealing with issues where this environment was slightly different. Yeah, yeah, yeah. Yeah. And some of those differences are so minor, it takes you days to track it down. Yeah. And it doesn't even seem like it should matter. I've done plenty of that. Oh, yeah. Every software developer has. Yeah. You know, the worst example I personally ran across is we had two servers that were at the same patch level, but the patches had been applied in different orders. Yeah, yeah, yeah. We had exposed some strange, you know, nuance of behavior that drove us nuts for days. Yeah. And that's really when you just kind of pound your head against the table and you're like, why do I do this? Yeah. There has to be a better way. And in fairness, today there is. You know, I mean, let's be honest, 15 years ago, this was a harder problem to solve. And, you know, you couldn't have identical environments because we had to buy these custom, you know, servers that were really expensive with these custom operating systems. And so, you know, we threw the app server on there as a way of maximizing return on that investment. And so, you fast forward to today and it's like, well, now spinning up a new instance is trivial. Yeah. And it takes seconds and it's free. And then in theory, there's also like enough isolation between your different spaces or whatever you want to call it that you could actually be running development and production in the same cluster. Sure good. Or you could actually, that's not, that's too fine-toothed way of putting it. But like, I always think like, you know, if you deploy to like Google or Amazon or Azure, like it's, there's, there's parity. I mean, I'm going back to the same thing of like, it's, it doesn't, it, that, that, that, that, that box that says, this is going to problem now. Right, right, right. The cloud doesn't care. No, it's like, hey, I'm going to run your workload. Yeah. Here you go. And so you, you sort of, again, you instantly get that parity without having to try too hard. I mean, I guess, I guess then what you have to do is to not mess up that default state that you get. And, and say like, I want this type of server in, in development. And then I want you to switch it over to this other type of server in, in production, which, or whatever they call it nowadays, profile.

And again, I mean, that's why that higher level of distraction is a win for us. Because if we're not down to that level in the weeds where we're checking boxes on, you know, what specifically we want, give me this many cores and this much memory or whatever, we have fewer opportunities to screw that kind of stuff up. And if we don't care, then we shouldn't, I would argue in most situations, like here, here's my workload, run it for me. You know, the cloud, again, doesn't care whether that's a prod workload or a test workload or a dev workload. And so that again, to me is fundamentally a win because then when I point my prod load balancer at it, same thing I tested in dev and the same thing my customers signed off on and the same thing my QA guys went through. That eliminates a whole class of problems. And I can make an argument that's one of the biggest advantages we get in the cloud right there is the fact that we get that, that parity. And then, and then like all these 12 factor things that sort of feed into each other. It's, it's at that point that like externalized configuration becomes really handy because, because in, in, in reasonable theory, so to speak. Like, right. Well, my test database better be different than my prod database. Yeah, yeah, yeah. Yeah. I mean, so that, that's the last thing we'll get to is like data. It's sort of like, oh, if I could have been stateless this whole time, nothing would have been broken. You know, our lives would be so much easier if we didn't have to deal with data. Yeah, yeah. But, but, but I mean, even before that, right. So if you have some deployment problems, if your configuration is externalized, again, not that it solves a problem, but it's a lot easier to like run the diff on it. Right. And, and then, and then if your configuration comes up as exactly the same, then I guess you're back to being screwed. But at least there's that chance to have like caught like, oh, we had this gigantic configuration file that is just a plain text kind of file that we can actually look at. Yep. Versus like you were saying, we manually installed these patches in this or in this order or not in this order. And then we had to update this thing. And like, so we don't have a way of actually verifying what we did other than our hopeful memories of it. Right. So anyway, so then the last part, so you got these backing services and like, like both, both in the work that you've done, right.

Like, and then also just thinking about them, like, like what is a backing service? Sure. What's the deal with that? Well, so, so I look at that, you know, especially as we talk about state. So we say these apps are stateless, but we don't really have stateless apps. You know, there's going to be something written out, but we need to have a durable place to put that, you know? So there has to be something that we're not just throwing away and recreating randomly. Yeah. You know, and all of the environments will give you some way of either having a file system you can write to or something that looks like a file system or, you know, database, things like that. And so it's being able to kind of compose that in as, hey, I need ability to write out the disk. I need the ability to save this into a database. Give me that. Well, here, here's a service you can plug into for my SQL instance or something. And I don't have to worry about how that instance gets spun up, you know, in a perfect way I just say, give me a large one or give me a small one. And it springs into existence and here's how I connect to it and now my app can talk to it. And that is also managed by my platform so I don't have to deal with all the rigmarole. So it's, I guess the way I look at it is we want the app to be stateless in the sense that I'm not putting anything in memory, you know, that I'm going to rely on coming back. But if I do need something to be there the next time this request comes in, I'm putting that somewhere durable so that when I come back, I can now talk to that. The new instance that gets created when this one gets sick or when we're scaling out. Yeah, yeah, yeah. Just gets pointed to that service that knows how to do it. Yeah, and then another way I interpret that is like, I don't know what people call it nowadays, but like don't embed middleware and frameworks. Sure. Maybe if the distinction is like a framework is just like a better string manipulator than whatever, that's fine. But like don't be responsible for running your own queue system or database. And, you know, don't be responsible for your own batch job management system. Like there's all these, I guess people call them services now. There's all these services that like you should not be embedding in your application.

Like, you know, going back to the thing I used to know, right, like if you're building a big Java application, even if it's in a jar file, you could like zip in a database, right? Like you could embed whatever you wanted in there and rely on it. And then you could even like if you had your own database in there, you could sort of like say, now I need storage. And then all of a sudden you're managing all that stuff. And so that seems to be another side effect of that is like you shouldn't embed stuff. Like you should have them running independently as a service, as their own thing. What do you want to have to manage? You know, and so by having that abstraction, then again, when a new instance springs up to either replace a sick one or to scale us out because we've got a big spike in demand, well, the database is a separate entity. It's over here and you just talk to it through this mechanism and it will store for you and it will retrieve you and whatever. Right. You don't have to worry about it. You know, if we try to get the job of tying those things back together, well, that kind of harms what we're trying to do here with being able to scale things up, scale things down, fix things, health heal, all that kind of fun stuff. You know, so it's just that abstraction becomes required if we're really going to get these other benefits. You know, if you're going to hard code that in, well, now that doesn't work out so great for us. So then when you were, you've evaluated things in the past, I mean, like, what, what do you do with the database? Because like, I don't think a database is a 12 factor. No, it's, it's not. And, you know, I mean, I admit that one of the hardest things we face, especially if we start talking about microservices is what we do with the data. Especially, you know, our traditional application was designed with the one data store and it had everything. And the monolith right wrote all over it and read all over it. And as you start pulling things apart in a perfect world, they all have their own database so that we don't step on each other. When I need to make schema changes, I can make schema changes and I don't got to worry about anybody else. And then you start running into all sorts of other interesting issues. Like, well, wait, this microservice actually kind of needs something out of there. So how do we, you know, how do I make that communication channel happen?

And, well, wait, maybe that isn't the best way to pull these things apart. So data, I still think, is one of those areas we haven't maybe focused enough effort and attention into. Yeah, yeah. It's weird that it doesn't get talked about more. Right. Especially because, again, we have always had it and we will always have it. I mean, that's kind of a requirement here in making our systems is, well, where's the database? I mean, I almost feel like at this point and maybe several points in the future, the answer is just sort of like, I don't know, it's that mysterious cylinder over there that runs, right? And, like, the best that we're going to do with our data is, like, there's still these priests and monks that run it. Right. And, like, we're going to get them to give us, like, a more friendly interface. Right. Right? Like, we're going to be able to send stuff in JSON and they can send stuff in JSON back to us. And, like, that's all the effort I want to spend on that cylinder. Like, there's not – I never – I very rarely come across, like – I mean, there's, like, all the children of no SQL stuff out there, right? Like, I was noticing – I think Mongo's web scale. Yeah. In our Paris office here, there's a data stacks office, right? Yeah. So, there's the Cassandra people and you got the Kafka people and – but – and I know very little about this, so I'll babble on and embarrass myself. But it seems like those – the problems with getting those things up and running are just traditional problems of, like, this is a big complicated data service. And so, you need to have good infrastructure and good system administration. And, like, at the end of the day, it's complicated, right? Yeah. Like, it's not this, like, simpler way of doing application development that comes and goes and – and I don't know. It's almost like the weird secret of cloud-native development is, like, yeah, you're still going to need those annoying DBAs. Or maybe cluster DBAs or whatever. Your data handling is always going to be – it's either going to be expensive and painful because it's just, like, some database, right? Or it's going to be weird and exotic. Yeah.

Well, part of the challenge I think we face in that database space is we have so much more choice today. You know, for a good chunk of our development experience, it's like, well, it's going to be a relational database. Yeah, yeah, yeah. Now, we have different flavors, but it was going to be a relational database. And then now we've kind of got this model where, well, it could be a relational database. That still might be the right choice in a lot of cases, by the way. But I've got this whole fleet of other choices that all are under this umbrella, NoSQL or whatever we want to call it. Yeah, I don't think anyone says that anymore. Probably not. It's probably too old. I mean, I'm just an old architect guy, so what do I know? But they all have different characteristics. Yeah, exactly. And what's fascinating is you see people will just glom onto one and they're like, oh, I love this one. Well, why? Well, because. And it's like, well, is that the right way to solve this problem? Yeah. You know, I mean, like you look at like Neo4j, you know, graphing database. There are certain problem domains that that is the perfect solution for. There are others where it's not the right solution. You know, I still think like we have a responsibility to know like when do I pick up which tool? You know, instead of just getting locked in on, well, I like this one because I like the name or I use it on a project and it seemed like it was fine. But does that, does that really give you the characteristics you need for this problem that you're trying to solve? Yeah, yeah. So that's, that's still to me the big mix. And again, the fact we have so many more choices than we did before makes it in some ways more difficult because you're like, like, can't just reach for one from column A. I have to figure out which one solves this particular problem. Yeah, I mean, it seems like a really weird space because like, or at least to me, because it doesn't seem like, like, let's call it data management. Is it data or data? I watch too much Star Trek. It depends on where you come from, I think. Yeah, there you go. It's like route versus route. You're state management, right? There you go. Like state management is this problem we've constantly been trying to solve and reinvent and we never really do. And in contrast, like when it comes to software development, like that's always going through wacky changes, right? Like, like we used to have whatever it is C was.

And then, and then we had this crazy thing of Java, like what that is. And then like all the Ruby people were like static typing. Oh my. And, and like, you know, all, and then, and then so quaint. And then the kind of like idea of, of like, whatever weird magic spring does with injection and introspection, all that kind of came along. Like nowadays, nowadays, for some reason, like all these like more statically type stateful things are like concurrency is a big deal. So like, there's always a lot of going back and forth and what's important in programming languages. Rediscovering something. Yeah, yeah. But, but in the state world, like every now and then there's these little brush ups of no SQL stuff that come up. But then eventually it goes back to what we're talking about is like, oh man, this is a really hard problem. Like, it doesn't seem like people have ever cracked the difficulty of running that long term, right? Operating it seems like the very difficult thing. And I think that's why we haven't seen that kind of churn there. Because let's be honest, like you lose the data, we got a problem. Yeah, yeah, yeah. You know, I mean, if you had a database where, you know, like one out of 10 writes didn't work, that'd be an issue. You know, I mean, if you, because I mean, so many of our companies now, they are just billions and billions of lines and databases. And if those go away, it's like, what do we have? I don't know. I mean, that's what it is. That's our customer data. That's right. You just have a stateless app. You know what? It scales really well and it's super fast. That's right. We just pipe it to dev and all. It's fine. And so I think that's part of it is there, you know, I mean, I'm not trying to diminish what we do as app developers by any stretch. But there's a responsibility there. Like, no, we can't lose data. You know, this has to be backed up. Yeah, yeah. This is an asset. You know, we can put a monetary value on this and we need to protect that. And so we can't be as cavalier maybe with that as we can be in maybe our process or in some of the other things that we do where we can just try these experiments and see what happens. Yeah. Yeah. And I mean, no CIO wants to say, yes, please experiment with my data. That would be wonderful. Yeah, yeah, yeah. That will get me a promotion. Yeah. No, like I was joking, CIOs don't want a stateless business.

No. Like, that's not what they're interested in. That doesn't work well in the P&L sheet, does it? No, no. And so, I mean, I know I'm repeating this that I don't know this domain, but it does seem like that would be one of the more attractive, or I should say the next attractive thing for, like, cloud services, right? And it's hard to, I guess at a certain scale for private cloud, it would make sense because you could centralize the data management and things like that. But definitely in a public cloud instance, and this is not so much true, but it does seem like potentially an evolution of easing the operations of a database in the sense of, like, as a developer, I don't have to operate the database. Like, I have to work within the constraints it gives me, but I don't actually have to, like, keep it up and running and make sure that the, I don't even know what people use, Fiber Channel, whatever. I don't have to know what the storage array is doing and, like, and what the interconnects are. Like, someone else handles all of that for me. Could be schools putting stuff in a tree, whatever. Right, right. As long as it works. And so there is, like, it does seem like if you make it easier, kind of ironically, to manage and bring databases up and down and all of that, then maybe there is some hope in making data or state, I should keep saying, easier. But I don't know. It's just such a weird, it's an annoying problem. And there doesn't, again, I don't know of, like, many people who, like, are fixing that problem other than sort of window dressing. Sure. Of, like, let's genericize a big honking database somewhere and then you can just kind of generically stuff stuff in there. And, I mean, the only thing that comes to mind is, I don't know, like, Google infrastructure very well, but I think they have, like, you know, two or three systems that, like, is, like, this is the database and you have to use this and we will run it for you. And, like, if you don't like the way it operates, too bad. That's terrible for you. Which is tragic because that's why we didn't like databases in the first place. Sure. Sure. That's why no one really likes DBAs because that's their attitude is, like, your query is nice, but you cannot execute it.

DBAs also have a thing against, like, vowels, right? I mean, you ever see, like, you hand them, like, your data model and they, like, strip everything out. Yeah, but they only have 32 characters. That's right. There's all these limits of things that they have to deal with. So, yeah, I mean, it's a hard problem. There's no getting around it. You know, and, I mean, for better or worse, I've largely stayed away from the data side of the house. I mean, I've been more of a front-end guy. You know, kind of like once the data gets to the server, it's like, well, somebody else's problem now, you know. And so I haven't had that kind of exposure there, you know. But it's an interesting part of this and it's something that I think we ignore to a certain extent in our own peril. You know, it is important that we think about how do we store this and where do we store it and what is the right choice for this particular problem. Well, we'll have to get our buddy Josh Long to do a new demo. There you go. That involves state. Yes, yes, we will. We'll see if we can make him do that. Well, when he's doing it, does he still do the reservation demo? Yes. Whenever he's doing that, I'll have to go up midway and pull the plug on his computer and be like, now what? Does it still work? Is that stored in a durable data location of some sort or another? Yeah, yeah, yeah. I mean, obviously, that's not the point of that demo. But, yeah. No, and in the defensive stuff we were talking about, I mean, I think that the evolution of how much easier and reliably it is to write applications, even though they're stateless. Like, that is, in my little career, it is remarkable that it's just, it doesn't take as long just to, like, write a reservation system. Agreed. Like, it's, and even more importantly, I think the thing that gets missed out a lot is it's one thing to be easier to write, which it certainly is. And nowadays, it's also easier to, like, run successfully. Yeah. To, like, not screw up when you actually want to run it. Like, it's easier to write it in a way that's, I don't know, DevOps friendly or whatever. Well, and to your point, this stuff is getting easier, and I think that's a good thing. And it's allowing us to solve problems faster and be more responsive and be better partners to the business side of the house. You know, and it is interesting. I was playing around with some boot stuff over the last summer, and I was, to me, being a longtime Java guy, I did feel like it was cheating because I had this little simple app I put together, and I had more lines of JavaScript for my front end than I had Java for my back end.

Yeah, yeah. And remember, when this was so much harder, you know, I mean, I'm old enough that, you know, Java in a nutshell book actually was thin, you know, it wasn't like this 900-page home. And, you know, but in the grand scheme of things, that's a huge win because I'm not dealing with some of that low-level stuff that I had to do 20 years ago. Yeah. And that's better for all of us, you know, because I don't want to deal with that low-level stuff. Now, what's fascinating to me is from the first time, we basically showed a business person, like, you could do that? And we're like, yeah, sure, we can do that with a computer. Like, the demand for what they want from us just keeps outstripping our ability to supply it. And we keep coming up with these better and faster ways of doing things, and the business just keeps asking for more. Yeah, yeah. You know, okay, can you turn that to 11 now? And we just keep reacting to that. You know, I don't know. I don't think we're ever going to find that place where we're, like, kind of sitting back. Yeah, no, that's a good, like, vaguely related last topic before we wrap up is, like, I'm sure you've experienced this, but it's always notable, I think, is the uncharged word you would use, that when you come over to Europe, and here we are in Paris, which I believe is located in France. I think so, yes. That's my understanding. One of the things people tell you is basically, like, well, we can't actually get rid of anyone who works for us. So anytime you're pitching lower headcount, that's not going to be effective, right? And it's almost like a challenge they throw out of just, like, you know, the punchline to lots of your business value jokes always ends up fire people, right, in order to achieve value. But, you know, I mean, this is, like, it always depends on your audience and how nice you want to be. But, like, just as you were illustrating, right, like, the goal is not so much to get rid of people, it's to, like, get each individual to do more work in the same amount of time. Or higher value. Yeah, yeah, no, exactly. That's what I mean. It's like, well, I mean, if you can speed them up and they, I mean, I guess I'm overthinking it. Like, if you can do five operations in an hour, then that's expensive versus if you could do 5,000 of those operations in an hour.

So speeding up things can bring down the value. But to your point, they can do more important things. Right. And I never like that phrasing because it sounds like, well, when does that ever happen? But it's more that you can do more things. Right. The business can expect more from you. And so, you know, you don't have to fire these people. Well, I mean, to your point, I remember building an app, God, 20 years ago now where, I mean, we were basically building the webpages with a bunch of print lines. I mean, that's really what it boiled down to. Yeah, yeah, yeah. Because that's what we had to do. I mean, that's… Well, that's sort of what it boils down to at the core, right? Well, that's true. Well, I mean, if we're really getting specific, it's a bunch of zeros and ones. The web server has this stream and then it just, it gives it to something. Right. And then you've got to write your junk into it and then it streams it back. You know, and so when I consider the amount of effort it took to put a web front end onto a relational database. Yeah, yeah. Take some data from someone and put it over here for storage. It was immense. I mean, it took weeks to do things that right now you and I could spin it up in five minutes. To me, that's a huge win because I'm not dealing with all these low-level, painful things that I had to do. Yeah, yeah. You know, I mean, I remember at one point early on in my career, I wrote a bunch of really weird JavaScript-y stuff to allow you to, like, edit stuff directly on the page and add rows on the page. And, I mean, it was an immense amount of JavaScript. You had the old XML HTTP request. Oh, yeah. This was before XHR. This is how old I am, right? And so the amount of JavaScript it took to do that was crazy. And it was because I wasn't very good at JavaScript at the time. Yeah, yeah, yeah. It was not the best code ever. Yeah, Netscape had something for that a long time ago, right? And, you know, I got lots of NANDs and all sorts of other crazy stuff as I slapped this thing together with bubblegum and bailing twine. Well, now you fast-forwarded it today, and I don't have to worry about that. I can just use React. Yeah, yeah. Or I can use Angular, you know. And a whole class of problems are solved for me.

I just grab this thing, and I'm working with a higher level of abstraction. Yeah, and you don't need a Netscape server to run JavaScript on the server side. You can run it on the server side willy-nilly. I don't have to worry about issues where it worked in Netscape, and it didn't work in IDE. Yeah, you remember when that came out, like Netscape was running JavaScript on the server side? I remember, that must have been in the late 90s, and I remember I spent a couple days reading whatever giant document they had. And one, I was very young, so it was very confusing. But two, I remember thinking, like, why would you ever want to do this? And then fast-forward today, and what's one of our most popular development of ours? Exactly. It's JavaScript on the server. Yeah, yeah. And there's a whole generation of us with graying beards that are like, why would you ever want to do this? That's right. All right, well, in a very non-creepy way, if people wanted to follow you around on the Internet, where would they go to? You can find me on Twitter at ntshutta, N-T-S-C-H-U-T-T-A. You can also find my website. Yeah, and you have... What are, like, your top two, three presentations that are recorded out there? Oh, geez, that's interesting. I think there's a copy of Sifting Technologies out there somewhere. Sifting Technologies. Yep. Okay. So, I've spent a lot of my career evaluating different things. Sifting. Yep, exactly. Sifting, which one of these, you know. It actually grew out of a talk I did a couple years ago comparing Angular to React. And after, like, the third or fourth time I did the talk, I realized that the more interesting part of this was not the specific comparison between Angular and React, but more the how do you compare... Angular and React and everything. Angular and React. Yeah. You know, because it was very reminiscent of work I've done, you know, throughout my career to compare X and Y or X, Y, and Z. And so, that's what kind of morphed into that. I've been doing this Cloud Native 101 a bunch, you know, which I think is kind of interesting. You know, and that just kind of gets into all the nuance of that, the 12 factors and all that fun stuff. Yeah, there's a good... You've seen a couple versions of that. There's a good, what would you say, depending on how you give it, like, a three to five minute discussion of each factor. Yeah, yeah. And the motivations and the consequences and sort of what they mean. Yeah. And we talk about microservices and we talk about serverless.

Oh, yeah, yeah, yeah. That's in there, too. You got to have some buzzwords in there, right? Yeah. That's nature of the beast, so... Yeah, someone sent me an email that they wanted me to come give a talk on, like, trends, which is always a fraught topic. It's hard to do that in a useful way, but... Yeah, I think, I mean, and that comes to mind because I think those three things, or there's four things you talk about in that talk, which is, like, cloud. Yeah, what is it? There's a trend, and then there's 12-factor apps, which is a trend, and then you have microservices, and then you got, like, are we allowed to say serverless? I think so. We can call it FAZ, if that makes you feel better. FAZ, okay, okay. Is that what that guy was saying today? I thought he was saying YAS, but maybe it was FAZ. Maybe, I don't know. See, I like architect as a service, except I don't like how the acronym is pronounced. That's the unfortunate part there. Yeah, that's a good joke. I used to ask, I used to joke about the three cloud asses, right? Like, it's that unfortunate thing that you have there. I think anything now can end in as a service. That's right. You know, we can probably get pizza as a service. So, do you have a third presentation you'd recommend? Oh, I've got a bunch of different ones. Let's see, what's another? Production Hardened Services is another one. Production Hardened. Yeah. So, that's one of the things we don't always pay attention to, and we have this crazy cloud environment with lots of microservices. You know, we've got all these spinning plates, and we confer an awful lot of trust on that service when we deploy it. Because it's going to interact with a whole bunch of other services, and potentially in non-deterministic ways. How do we make sure that we didn't just put something in our environment that's going to bring the whole thing down? You know, so it's what kind of stuff do we need to be thinking about as we enter this era of not one app that we can step through in a debugger, but five, ten, fifteen, a hundred services that are all interacting in unique, interesting ways that we may or may not understand. Yeah. So, it gets into a lot of the things that we need to try to do this in a way that will maintain sanity. I mean, clearly, unlike those monitoring and log dummies, you're going to need some advanced analytics. You're going to need some monitoring and some logging and some alerting and a whole bunch of other stuff. But it really tries to drive into how do we have this healthy microservices biome.

You need some adaptive baselining and application tracing. Word. That's what's going to save us. Well, as always, this has been Pivotal Conversations. If you want to get the most recent episodes or peruse our back catalog, because you have a few things better to do, you go to SoundCloud.com slash Pivotal Conversations. And increasingly, not on Thursday, but it's traditionally on Thursday. We post the full show notes for this. I'll see if I can round up some of the talks that Nate mentioned here. And I don't know. Maybe I'll find some old 2002 literature on how, oh, I know. IBM's autonomic computing was like the height of the advanced analytics. It's a really good paper, if you don't remember that, from either 99 or 2000. And you could just kind of like shift it into the way we talk about things now. And it's a little delightful. It's kind of like the flying car of cloud. Sure. I don't know if that's a good analogy. Anyways, yeah, show notes. You can find those there. And with that, we'll see everyone next time. Bye-bye.

Tanzu Talk · 🍕 Get more pizza.