Here's something I wanted to ask your advice on. So you're up there in Seattle, as I'm fond of reminding people. And maybe we have several other guests in the room. Maybe they can chime in. But I was at Costco recently, and I bought a new bag of coffee, which was roasted in the Cuban style, except they said that in Spanish, which I should probably be able to pronounce. And I looked at it, and it was espresso coffee. And now I am not making espresso out of it. And what I want to know is that if I'm making regular brew coffee, like, how many years am I taking off of my life each time I drink several cups of that? Like, is there something about it being espresso coffee that's, like, ruining me? Like, do I need to just not drink those two bags of coffee? What would your Seattle friends say? I mean, I'm sure it's, like, smoking two packs of cigarettes a day or something like that. But, you know, believe it or not, I'm whatever, 40-ish, and I've never had a cup of coffee. What? No. What? I'm dropping a truth bomb on me right now. Wow. You don't all have to make any kinds of confessions. No, I don't really like hot drinks. And so at this point, I've gone this far, and at this point, it's only out of principle. Like, I'm halfway to death. I might as well finish it. I tried drinking coffee when I used to, I worked construction artists in high school. But I had to be, you know, at the work site at times, like, 4th, 3rd in the morning. So I get there, I try to drink coffee, and I'd be like, this tastes like crap, and I'm still tired. And then it's just like, you know what? This is not worth it. Yeah. Just, yeah. Coffee just doesn't do anything for me, so. Yeah. That is a hurdle. Killing you quickly. I drink decaf. Okay. Wow. Wow. I asked the wrong crowd of people for advice on this. Kenny. Yeah. Kenny, come on. Save me. I drink coffee. I don't have an answer. Yeah. He drinks motor oil. So I don't know. Yeah, he goes over to that cold brew stand and just, like, wraps his lips around it and pulls on the lever, which, uh... Take stamps. I've tried to tell him that's not hygienic, but I just keep chasing around with one of those little Purell bottles. And he's like, what? It's really funny. I don't drink it. You gotta get, like, one of the little water bottles you use with cats.
2:00No, Kenny. No. Why don't we introduce the people we have as guests on this episode? You're in the room with them, Richard, so why don't you rattle it off? I mean, don't introduce them, but, you know, point to them so they know which order to go in. I'm getting the wave. Um, this is Dormain Draywitz on the, uh, the product team, product marketing team. Thank you. Here at Pivotal. I'm Steve O'Grady. I'm from Redbook. I'm Kenny Bastani. I'm a spring developer advocate here at Pivotal. And then, as always, we have Mr. No Fun, who doesn't drink coffee all of his life, Richard. Just, it's such a letdown. I don't know. And, and, and, and myself, Cody. So I think, uh, you know, so Steven wrote a, a post a while ago, which, which I think I have to say for myself, sort of like, uh, typed down, so to speak, something I've been thinking about a lot is like, uh, people don't talk about like data and databases in, in the context of, uh, DevOps very much, which, which I think is, is sort of curious. And I think, I think what with the collection of people we have there, I think we could have an interesting brief conversation about that. But let's start off with, uh, with you, Steven, can you give us kind of like a, uh, sort of an overview of like what's up with databases and DevOps? Yeah. The Reader's Digest version. Sure. So basically the idea is, is that we spent a lot of time looking at, uh, a variety of technologies and application development processes and just anything that a developer will use, uh, to build and construct applications is something that we're interested in. And what we've noticed, you know, certainly over the past, oh, I don't know, at least a couple of years, you know, say four or five years, um, the application development processes evolved quite significantly, you know, uh, you know, through the introduction of lots and lots of technologies, you know, from, uh, things like decentralized version control, all the way out to sort of modern CI/CD systems and pipelines and everything else. And, you know, the funny thing is, is that sort of, as you, as you noted in lead in, when you look at the way these applications are developed, the database is basically completely absent, right? You know, nobody talks about the database, nobody talks about the DBA.
4:02So, you know, one of the things that, you know, we're trying to figure out is, you know, is that appropriate, you know, is there an opportunity, uh, you know, potentially to try to bridge that divide between application developer and DBA, you know, very much as we have seen the industry do, uh, between two previously distinct categories in dev and ops. So that's kind of the idea that I don't. Did you, why, why do you think there's that divide? Is it because it was the same divide we had with dev and ops, where it's like, I don't like dealing with that team. So therefore app teams said, I don't want to deal with DBAs. So we'll just ignore that and ship software itself in its own way. I think there's a variety of reasons, honestly, some of it's structural, some of it's cultural, um, you know, and some of it's just sort of a matter of focus. Right. So, you know, taking, you know, taking a couple of those, uh, in many cases, it is a, uh, you know, very much sort of a cultural divide in the way that dev and ops was. Right. Um, all the scenarios, uh, when I was an SI and, you know, sort of, uh, uh, on client sites and buildings, um, I had to deal with ops. I hated ops. Um, I had to deal with DBAs, uh, I hated them slightly less, but they were a pain in the ass to deal with. And, you know, vice versa. I'm sure they hated us, you know, because we wanted to do things and then sort of move quickly and do basically, you know, things that didn't work for them. Um, so there's that, I think that's part of it. Um, particularly for many developers to, uh, any developers ever been on call is probably not super fond of ops cause they're the ones who typically call you in the middle of the night cause you're broke. Uh, so you have, you know, sort of those issues, but also just from a structural standpoint, right? Many of these very much like dev and ops were, were separate entities, you know, sort of, you know, Hey, I do my thing and you do your thing. You know, we don't really talk to each other. That's structurally been the way that many organizations have been set up, right? In other words, you have DBAs, uh, people responsible for securing, protecting, you know, designing the scheme is associated with a particular database. Um, and then you have people who do the app development. And to the extent that those are involved, you basically will say, here's what I want my application to look like.
6:02Um, figure the database out. And then that's kind of it. Right. And you know, they're done and, um, they're just not sort of integrate, integrally, uh, involved in that process. What do you think makes up the data DevOps story? I mean, I think there's, you mentioned some things like source control that became a big part of the dev life cycle. And, but I mean, automated builds and a different approach to maybe service abstractions, data replication, caching. I mean, what are those things that you think to put, bring it tangibly, if I'm a DBA listening person's going, what do you really mean when we think of DevOps and databases and everything from blue green deployments to schema changes and source control and CI builds, bring this down to a. Sure. Tangible level, right. Uh, yeah, I think there's a number of different things. So when you first, you know, when you design an application, right, you know, think about, let's take something simple. Let's say you're a rails developer. All right. How do you set up, you know, the, the database typically you're doing it in the YAML file or something like that. Um, what a lot of applications find out sort of as they grow up is that that maybe isn't ideal for what you want to do. Um, maybe you need to scale, maybe you need to shard it and so on. And all of a sudden you're out of your depth. Um, all of a sudden you're out of your expertise, right? So having, you know, deep gaze intimately involved in that developer process can help you either at the outset or as your application scales up. But there's also lots of sort of trivial things that we don't think about, right. In terms of, oh, uh, you know, as you're developing sort of an application, you know, one of the things that people can sort of gloss over frequently is test data, right? Where do you get that test data? How does it, you know, how is it generated, right? In many cases there are, you know, there are vendors out there who do nothing more than generate essentially the sort of lorem ipsum equivalent for, um, you know, that kind of data where DBA in many cases can basically say, all right, I can, you know, fork a copy of our database, salt it sufficiently. So we're not exposing anything we shouldn't be. And boom, now you have essentially real live test data that is a safe, but B is going to work, you know, basically going to give you the opportunity to test and build your application as if it was something in production.
8:02So I think there's lots of different ways this can play out. Frankly, we, you know, we can think of a couple of them, but, you know, very much as we've seen with DevOps, where this has taken lots and lots of different pathways, uh, people, you know, involved at the early days could not have anticipated. Basically, I think it's the same thing here. Dormain, you, uh, covered a lot of stuff with Greenplum as well, as we think of just beyond the database to the data warehouse. I mean, is this a complete, I mean, if you were introducing this to a Greenplum audience saying, Hey, it's time to start doing some CI and CD to your data warehouse and artifacts and source control, would you be, you know, burned at the stake or is this something where that kind of crowd you think would welcome this idea of maybe a little more rigor around the maintenance and the setup and the on-demand nature and all those things around warehouses? Well, I mean, like when you're looking at the data side of things, like, you know, jumping to the data warehouses, now you're talking about something that is more than a couple degrees removed away from the developer. Sure. I mean, there's a lot of stuff that's a lot closer to the development teams that would probably be a more logical starting place. And I think as those things, you know, potentially evolve and change in this direction, right? And to the hypothesis you've got, then that, that might have like downstream implications for teams like the data warehousing team. I mean, right now, like what's kind of interesting that, you know, we're, we're seeing a lot of already was, you know, the, the data that people want to take a look at and sort of apply data warehousing to is sort of, it's all over the place. Right. So different teams just went down different paths. You know, there was a lot of like, okay, we're going to create a Hadoop cluster because that seems like the cool thing to do. But some data is going in there, but it's not necessarily usable in the way that we want it. And then, oh, this team over here is building stuff in Amazon. So they're, you know, they've got a bunch of data accumulating in S3. And so, I mean, the last year, Greenplum actually started to, I mean, not started to, but took
10:03some additional steps down being able to just read from all these different places as external tables, because I think the notion of, okay, we're going to create a data lake and it's going to be one data lake to rule them all. And like data will flow in and unicorns will flow out. It just has been proving over and over again to not happen. And so it's for various reasons, right? Like people, different parts within an organization don't want to give up their data, right? The data huggers. And they've got, they're like, no, we're special. And, you know, they're, they're putting it into Hadoop. Like there's a shortage of skills to be able to do anything with it. So the natural, you know, gravity of data is just not actually that focused. But it seems like it's also coming to the developer more, Kenny. I mean, as you look at things like streaming data and Spring Cloud stream, and like all of a sudden we're getting closer to the warehouse than we probably have been before as a dev, because I might actually be streaming data from something front end to that. Or do you think this is forcing the issue finally? I mean, the things that Steve's bringing up here, because devs are now integrating data in different ways and moving data in different ways, they can't just stand back and say, eh, the database. Like that's a big part of their app and their service or microservice or whatever, right? Yeah, so this is a really hot topic, kind of near and dear to my heart in a way. I was a developer evangelist over at Neo4j a couple years back. And this is something that we talked about a lot is what is the future of the database really look like when you're going to a cloud environment where you have all these application components, infrastructure that's ephemeral, but that data isn't ephemeral, right? That data has to be somewhere at one time, right? It has to have backups and there's a process there and you need expertise. And so what I found is the DBA really has an opportunity, and this comes from personal experience developing microservices, that the DBA kind of becomes a little bit more of an architect. The developer is still doing the work and they should be competent enough to do that work, but the DBA is there to do reviews with the developers and make sure that the standards are being set forth in the application development process.
12:05So that's what I found. Yeah. So you're seeing that come closer. You know, I had a question for Dormain. I think, like as you were describing the world of data people, I don't know, I'd be interested what Kenny and Steven think if this is the same calibration that I was having. It's like, oh, structurally sounds like similar problems that the apps, the app dev and the ops people have, right? Like you're sort of like, uh, you're sort of like chasing all these trend things and you got to worry about things actually working and all of that. But, but I'm curious, like, I mean, ultimately a lot of what we're shooting for with all of the, like a few episodes ago, we talked about what cloud native is, but it's sort of like, I would like to more quickly innovate and do things right. Like currently the way that I'm running it doesn't move fast enough and I can't do enough with it. So I would like to change things around so that, uh, I can change stuff up. And, and when, when you look at like what DevOps has been doing a lot of what the application developers and the operations people have been figuring out past constraints they had in their life that caused them to need to move slow. So for example, uh, usually when we deployed a production stuff breaks, so we're going to try to avoid deploying to production, which obviously means you move slow. And I'm curious similarly in the data world, like what are some constraints that data people have that they would need to like figure out if they want to move fast. And, and, and as an example, cause my questions are often statements and very confusing, but like, you know, I often hear from data people, like you can't lose data, right? Which, which seems like a simple answer, but if you think about that mentality in the application world, I mean, like you can kind of lose an application cause you can just restart it. Right. So it's not like such a big deal. So like what are these kinds of constraints that data people have that are almost like these huge constraints that change how they think about how fast they can move and how devopsy they could be. Yeah. I mean, data loss is definitely one of them, but that's also kind of to that point of, you
14:06know, where in this sort of the spectrum of data practices, you know, when you're, when you're dealing with something like a gem fires and other product that I've worked on, that's, that is often in a position where like data loss is not an option, right? It's, it's a try, it's doing something that's transactional, et cetera. Um, whereas like certainly data loss is not acceptable, you know, for even something like a green plum, but it's much more tolerant to downtime. Um, whereas something like gem fire is not tolerant to downtime. Like what it's, it's doing, doing something that you've got to keep it up and you can't lose any data. So there's just some really tight constraints, but there are other systems that don't. So it's not the same as what I'm trying to say. It's like, there's, there is a spectrum in data. Yeah. Um, and there was something else in the first part of your question slash statement that was confusing that I had another comment on, but you've managed to totally. Wrap you up into a pretzel. Yeah. Like it was like a weird Jedi mind trick or something. I don't know. Um, um, so yeah, come back to me, but it's, um, the, the notion of what Kenny was referring to in terms of bringing, bringing the DBA in as that, that consultant architect and the notion of like, we've sort of touched on this with a couple of teams, like the balance team, right. And how do you start to incorporate data on that balance team so that they're, uh, they're present for not only like decision-making, but also for the discovery of requirements. Um, so, you know, like this is one of our data scientists, Ian, uh, Houston or Huston, or however you say his name, he's British. So I don't trust any pronunciation as soon as I start thinking about people from there. But nonetheless, um, he, he actually wrote about this and, uh, we, we interviewed him on, on the pivotal insights podcast.
16:07And, you know, he, he talked about a lot of really interesting points from a design perspective, which is like, Hey, to be sitting there in the user interviews that like the agile team is doing is amazing from a design perspective, instead of just getting like pitch something over the wall to be like, Hey, we need a predictive algorithm for this. But it's like, Oh, it really helps if I get to sit there and actually ask the user some questions about like what they're trying to do. Cause that's going to change maybe what I go do. Like from a data science perspective, which probably also ultimately influences what's going to happen for how are we going to structure the database? What database are we going to use for this? Um, but yeah. I want to piggyback that you may ask the question about constraints that kind of might impact this. I want to ask you, Steve, are there other constraints around people who like throwing logic in the database? And does that, does this change in DevOps where it says maybe I don't do store procedures or I don't use views or user defined functions and maybe even some things that exist in NoSQL databases. Is that a constraint that teams have to actually change their behavior or do you just have to adjust to using those things in a DevOps fashion? I would imagine in many cases you would just adjust. Um, I'm sure that there are, I'm positive that there are exceptions where in other words, it's like, like you crammed this full of so much procedural logic and so many stored procedures that it just is not, it's not going to be compatible with essentially a modern, you know, life cycle, you know, you know, particularly as quickly as we move and so on. But my guess is, is that in, in a lot of the cases, you know, you can, you can basically adapt. Um, and in some cases, you know, that may be pulling things out. Like I've had to do that before, um, years and years and years ago, pulling stored procedures out because, uh, the way that we were doing it in the content management deployment, um, was not compatible with some new operational standard they had, you know, for the data center. So, you know, obviously, you know, infrastructure people will have a history of, um, uh, dealing with changes just like this. Um, but yeah, I think I would, I'd be willing to bet in, in a lot of cases you can, you know,
18:12basically deal with that kind of logic. The only other one of the constraint I wanted to bring up for Kenny was, you know, that question of data loss. And, you know, we all love in this world of app CICD, this idea of like, I'm going to ship 10 times an hour and I can do zero downtime because cloud foundry. You'll do blue green or I can do other ways to do rolling updates. What does it mean in a data world? How in the world do I do continuous delivery of a database where there is actually schema and state? I mean, what patterns do you see for actually not losing data or not taking downtime and actually swapping a schema or things like that? It's a great question. Um, and it's one that kind of keeps me up at night lately a little bit, especially recently. Um, I think a lot of people here probably heard everyone about getting the labs data loss. Um, now when you think about that, the microservice level, right, you have all of these independent agile teams who are delivering continuously into production and they're delivering features. They're delivering business logic, right? Um, so they're adding business capabilities to the whole, but they're all integrated together and each might have, and this is the pattern that they have their own database and the developers are managing that database, um, in a lot of cases. But what happens when one of those microservices that's has, um, relationships to other services, uh, in data there and they have data loss in one service and you can't get it back. But all of the other data was added to the other services. Now you have kind of an inconsistency issue, uh, where you're not able to recover without pulling everything back. So that's really, I think a problem that, um, most people should be worried about. And I think that DBA is a really, uh, kind of critical to this process, or at least on the platform side to make sure that they're handling these concerns across the board, uh, to make sure that this doesn't happen. Have you seen solutions, Steve, around that? Do people, again, do that work? It's people skittish and that's why they do stage, do they do plan deployments? I think the latter. Yeah, I really do. I think people, and this is this, you know, kind of circles us back around to the original point. Like so much of, um, you know, so much of we see, you know, of what we see today, you know, from a, from a model standpoint, from a procedural standpoint is due to the skittishness
20:12or the concerns around data, right. Um, in terms of, well, okay, you know, it's kind of hands off and I don't really talk to those people and I'm not really sure how to, you know, sort of, uh, um, you know, go back and forth. So yeah, I would say it's very much the latter. Is that where service abstractions or caches come in handy to insulate me a little bit from changes underneath? Like I can do V1, V2 of services that have different schemas and maybe I don't remove fields. I just make them additive. I don't know. Have you seen things domain where a cache or some other abstraction protects me a little bit from these database level changes? Yeah, definitely. We, I mean, we've seen caching come up a lot as customers are sort of going down a, Hey, we want to refactor an application on Pivotal Cloud Foundry. And, you know, this is going to lead to potentially a lot more calls into a legacy database. Oh, wait, that legacy database is a mainframe and it's really expensive or, you know, to have all those calls or there's going to be performance constraints, what have you. And so putting a caching layer, uh, between, you know, the, the sort of refactored or in process of being refactored application and that legacy database sort of suddenly helps lift some of those constraints and, and lets those, you know, new microservices call to something that's caching that data. They can, they can they can, but it's not going to stress that legacy database, which folks can still continue to run their legacy processes around, et cetera. Yeah. I think it's, it's interesting to see too, where you, you begin to have, it's certainly not a, a common, uh, practice today, but you do begin in some scenarios and, you know, some vendors and, um, uh, some organizations begin to have a notion of forking databases, right. You know, be able to say, okay, we're making a decision. We're doing something at this particular point in time and, you know, basically, okay, we're going to fork it. So we're going to take a copy of this database. We're going to go down this road. If this proves to be the wrong path, fine. You know, we have what is effectively a backup. Um, but in other words, it's, you know, very much a notion of, you know, trying to take a software development like approach, uh, you
22:19know, kind of with data, right. You know, having the ability to essentially have different versions and fork one and okay. Um, you know, how do I reconcile these things if necessary after the fact? So it's really not there. It's really not common. You very, very rarely see this in the wild, but it is a notion that is there, you know, has been sort of floated in, in certain corners and I would not be surprised to see that gain currency over time. Yeah. I mean, that's, I mean, and I'm starting to see that as I, I talked to a number of the ISVs that are building integrations with Cloud Foundry and there's a, there's a range of what types of integration. I mean, the most basic is usually, Hey, we want to have a, a service broker so that people can just connect to an externally managed version of this, um, database, right. Because they're, there's, they're already existing and we just want to be able to help developers get up and running. Um, maybe we were going to give them sort of more privileges and capabilities in terms of, Oh, create a new table or something. Um, but there's also, I don't want to like steal any thunder because I know some of these guys are going to be making announcements soon, guys and gals, but yeah, looking at ways to, you know, clone databases, make them, you know, discoverable, um, as well as sort of making them, you know, Hey, you want a new one, you can have it on demand, right. Fully managed, uh, type of thing. So there's, there's a lot of interesting work being done, but it's definitely, that's all as far as I can tell pretty early days, at least from my vantage point. That's a great point. I think just to add what Steven was saying, um, I think that there's a place in this microservice cloud native world for change data capture where every time you're writing some of the database, you have some kind of replica set that you can go back to. That's not really a database back. It's not like a monolithic backup, but it's a way to stream to somewhere else where you can keep that data safe. Um, that way, if you experience data loss, there's always going to be something that you can go back to to restore. Yeah. Sorry. Historian or just even having a good, I don't know, event sourcing model or something there. Yeah. So one thing I want to add one more question, I guess each of you get a shot at it.
24:21So for people who are trying to think about, all right, some of this sounds really interesting, science fiction, future stuff. How did, how did the company start to get started with some of this? Is it putting their database scripts in source control or trying to make it actually possible to get a database on demand? So a dev doesn't have to go through hoops to just get a local copy with some anonymized data. Like how do we recommend that a company does try to get started with treating databases as a little more of an agile asset versus these static things that we're terrified of domain first. I mean, I, I think some of it comes down to the people. So the notion of the balance team and bringing DBAs, data scientists, analysts, whoever is going to ultimately need to work with that data, like into the team that's building the application where the data is going to be produced. I think that's going to have, that's going to be some part of it. I agree with Dorman 100%. It starts with culture, right? And then everything else comes back to that. I think that's the first step. Yeah. So they stole my answer. I was going to go with culture. You know, basically integrating the teams as quickly as you can. And, you know, as patiently as you can, because it's not obviously an overnight process, I think is important. But beyond that, you know, really, I think one of the things that would be, you know, helpful beyond just sort of the initial association is really trying to, you know, get a, get an understanding of the data side of your business, right? Because, you know, many of the people listening to this probably have quite sophisticated, as we talked about, application development, you know, processes and policies. And you'd be very useful to begin not just sort of integrating teams, but taking, you know, having architects or people from an app dev standpoint, at least think about, and again, you can't parachute in and start telling people what to do with their data, but at least start thinking about, you know, from a, from an outsider's perspective, right? All right. What are the kinds of things that we've learned building applications that can be applied to this world? Because obviously they're apples and oranges, right? It's not as if everything's going to transfer, you know, with full fidelity, but there'll be a lot of lessons learned, which is, you know, hey, this is what we used to do in sort of building applications and waterfall.
26:25And this is dumb. Like we just don't do things this way anymore. So, like I said, I think trying to, trying to extract and distill some of those lessons learned, you know, and impart that. And again, how you do this is critical because if you parachute in, like I said, and start telling everybody what they've been doing wrong for years and years and years, you're not going to make many friends. You're going to set your effort back. But if you can do it diplomatically, if you can begin to have that conversation with the data teams within your organization, I think you, it can at least kickstart the process. Because what you want ultimately is for the data teams to start thinking about, cool, you know, hey, the application development, men and women have been doing a great job of, you know, sort of modernizing what they do and constantly innovating and so on. You know, this is cool. We can do the same thing. You know, you really want them bought into, you know, how do we make our lives easier? Because that's ultimately what all this is about. We want to ship things faster. We want to do it with, you know, sort of not just better velocity, better efficiency. So we're not going to call in the middle of the night and on the data side that you can move just as quickly without losing data, right? You know, because that's the key here. Well, speaking of making people's lives easier in the middle of the night, if you haven't already subscribed to this podcast, what you should do is find the RSS feed over at soundcloud.com/pivotalconversations. And you can just subscribe to it and it'll automatically download. Now, I would recommend that you go to all of your devices and subscribe to it there so we can goose our download numbers. That's always appreciated. But if you just want to see the show notes for this, you can go to cote.io/conversations, probably 53 or two, just sort it out. And also, if you go to pivotal.io/podcast, you can see a list of all of the podcasts that we have, including this episode. And it's always great if you want to leave a review in iTunes or recommend it in Overcast. And with that, we'll see everyone next time. Bye bye.