Databasing with Greenplum, with Ivan Novick

Episode 93 · 46:03 · SRT · archive.org · all Tanzu Talk

So why don't you introduce yourself yet, because I had a few questions right off the bat. Hey, yeah, I'm Ivan Novick. I'm at Pivotal. I'm a product manager who works on Pivotal Greenplum, which is a data analytics database used throughout the world for storing and analyzing big volumes of data. So I was watching, I think it was one of your videos. I was watching a video essentially about Greenplum, but then it turned out delightfully to be History of Databases. You did that one, right? Yeah, yeah. That was awesome. Yeah, I've done that a few times. So the question I had, I mean, just to boil it all down, like it was fun to see, like, you know, you got the, I'm going to mess it all up. But there's basically like, you know, the Apollo mission has a database and then there's IBM and then there's still IBM, more IBM, very IBM. It started with the census, the U.S. census back in, with punch cards, right? You had punch card computers and the first usage was to add up all the people in the country, in the U.S. and put them in some sort of a calculating device. That's right. That's right. And so to my question, which at some point you went over, but it's always curious to hear, is like, when did the jump over, well, let me phrase this another way. Why do you think it is that, like, the relational database became, like, the king of databases at some point? Like, what about it and the time that it happened was sort of like, and what I mean by king of databases is, like, I feel like maybe by revenue, by number of instances that were out there, by people using it, by simplistic ways of saying it was the king of databases, that it was sort of, it's for many years, it probably still is, was just like, that's what people do. In the same way that you would say, like, the combustion engine is the king of locomotion. Relational database is based on set theory, relational algebra and set theory. So, you know, at a high level, you've got a set of data and you want to reason on it in

terms of overlaps, intersections, unions, you know, you think of your Venn diagram, right, in terms of, you know, I want to, you want to select and understand the data grouped by the data. And it's a, it's a natural way to, people want to think about and process the data. Is to, is, as opposed to, for example, programmatic processing of data, where you say, you know, I'm going to write a for loop and just write a program to process data. And you say, while something, I mean, that's a very kind of cumbersome way to, to deal with large set of data. So a more natural way is to say, okay, here's a table of, of orders and a table of products and a table of customers. And I want to understand which customers overlap, which orders, which overlap, which products. So fundamentally that was brought up by EF COD in the seventies or eighties. And it's stuck, right? It's stuck as a way to, to store and query the data. I mean, is it, is it sort of like, was it driven by like cost or performance or maybe even like cognitive knowability? And, and, and I asked on the third, cause like, I remember when I was learning relational databases and like, it made no sense to my feeble human mind. And more specifically, someone who didn't know set theory very well. Like it was just like this whole idea of like joining, putting stuff together always seemed very like weird. So it's like, and then, and then there's the whole like object relational mapping thing, which is this whole sub industry, which is like the translate between object orientation and like relational databases. So like cognitively, a relational database is like its own island of fun, at least from my perspective. And so like across those three characteristics or maybe a fourth of like cost performance and like a knowability, like, I'm always curious, like why that became the way of doing things

instead of like poet or whatever object or in a database was out there, which seemed like cooler. Well, and, and poet came and these, these object databases came after I think relational was already established, but, um, and, and, and obviously never got, um, critical mass or traction because. Right. Right. And, and, and, and I guess, I guess I bring them up because whatever that initial power of the relational database with was, I guess, I guess something can clobber itself by being insufficient, but I feel like the relational database was just sort of like pushed all the O databases aside and was like, no, no, no, I'm still going to rule the nineties. Sure. So, I mean, so, so the sequence was, there was something called some sort of a hierarchical data, hierarchical database, um, that, that IBM was pushing. So, so, so basically the first thing was, um, the first fundamental thing was the electricity, right? The electric computer. So we went from punch cards to electric computers and the first databases on electric computers were, um, obviously from IBM because IBM was dominating computer industry. And so, um, they were, there was a system called, I think IMS was the name of it. And it was a hierarchical database, which to be honest, I don't know too much about because it's basically obsolete right now. Then, so the reason relational came in, uh, as I mentioned before was that it was, it had a kind of a mathematical, um, backing and it also was business practical. So from a, and I think the key thing on the price performance, all these things was, was to be honest, the usability and the, the, um, kind of from a business point of view, the ability to ask a question and really not write any software but just store the data and ask the questions that need to be asked to generate business reports so it mapped well to to business

and when it maps well to business then that drives economics right because there's only so many people who want to buy a huge database not for business right yeah yeah and and then and then uh like like i mean maybe the guy i think the point you you make good is it's sort of like it's useful yeah right and it's it's it's easily useful the usability on a tie and then and then my last like uh you know walk down the uh the ivy encrusted memory lane path so how how would you describe the relationship between spreadsheets and relational databases right because it it seems like there's a uh i don't know if it's a carton horse and a chicken and egg type of thing but they seem like very closely tied together i think they are yeah they operate they are i think they are very closely tied and microsoft excel probably economically is equally successful as any database commercial um it doesn't it so the fundamental so so okay it brings to the one of the key points about the database business which is it's extremely hard to implement and there is no solution that has been created by any humans that is the ideal database yet right every major database product has clear known um compromises made whether it be around scalability or or mostly around scalability and how they do trade-offs so the trade-off excel makes one of the main trade-offs excel makes it's a single user system right i mean yeah sure you can share the file but you can't really have 100 people edit the same spreadsheet at once well i guess with google spreadsheet that changed but not um not atomically right if you want to remove money from from one bank account and edit to another that's one of the principles of relational databases which is the acids the the that when you do a

transaction it's guaranteed to either happen or not happen right so you can you can account you can have an accounting of the world you can say this is the current true state and with locking you can have the different people update the data and so if you wanted to just create a single user database that can pretty much be done in a college project right but once it becomes a billion dollar investment is when you want to have concurrency and scale so excel is super useful for a person who just wants their own to look one person to look at some data um it doesn't really have easy ability to do let's say um let's say you had one spreadsheet of orders and one spreadsheet of customers and then you say okay i want to see give me a report of the average order size per customer i mean it could take a while to to make that come out whereas it's one line of sql in a database right yeah that's true i i think i think i spent many late nights trying to understand the theory of pivot table just like i still don't know i still don't know how to use pivot table exactly anyhow uh we'll come back to uh to uh more database stuff talk about green funds before that uh there's just a few little news items to go over now you know there's there's uh around here we had uh we had a big relief pivotal cloud foundry 2.2 now uh i think it was jared who wrote this stuff right like i think i think i still i'm going to set aside four hours to read that blog post which which i mean more of which i mean more of a complimented way of like uh there's a lot of stuff on there and uh it's very well documented but there's a there's all sorts of uh under the hood i don't need like one of the i was trying to pick out one that's notable and i think uh having more detail and controls over auto scaling is always nice because that's uh one of the immediate useful

things of of doing anything cloud related is having elasticity and automation uh something like scale so let's see that we have more improvement there but how about how about yourself richard if you were to pick out one or two things to highlight what what are your favorites in there yeah i'm glad you like jared patented a giant list of things blog post structure for announcements but he makes good use of headings and pictures that makes it very easy to read so that's a patent pending i was i was i always awaited the uh back in the early 2000s when i was a programmer i waited each release because they would have like 20 page release and it was sure i've been working to read the release note yeah so this is a good one as you mentioned i think the auto scaling stuff which uses this new log cache subsystem for for reading logs but i think what's pretty cool about the auto scaling is yes you can do custom metrics but there's even some smart things on comparisons like hey if the maximum is different than this you know other minimum and both may be custom metrics you know like let's say the you know the current is approaching the max maybe that's when you auto scale so it's not even just like hey this number hits some sort of threshold maybe it's actually based on two different values coming from the app or from the service so some really cool stuff coming available now in auto scale and that it's a in pivotal web services as well that's always where you can see these things before it hits the the software itself so that's pretty cool uh the other thing that jumped out to me i think was bosch and credhub support so bosch credhub is generally available the runtime credhub is available this has been a component a part of cloud foundry for a year or so now but it's kind of continually been beefing up it hasn't been just released at one time so now credhub has hit a really nice mature point storing credentials securely for your apps for the different credentials within cloud foundry itself so really neat to see that available and then just some some good infrastructure things supporting azure stack formally supporting things like deploying to multiple v centers as part of like

an az and high availability architecture that has been there before but there's been some nice new support for supporting multiple data centers on v sphere so getting these multi-site deployments is tricky so we want to make sure that we make that simpler so some good things in there definitely read the giant release notes that we'll add in the show notes but good release we keep marching forward with something good every quarter and it and speaking of releases although in a good way i i haven't had time to look at it but what do you think about uh like at any given time there's like six percent problems happening in your life computation-wise oh i'm sorry i'm sorry 6.1 percent chance of breaking something in a library yeah it's about accuracy Coté 6.1 percent uh so i had uh it's funny sometimes i purposely think about you know hey if i tweet something at you know eight o'clock on a week night pacific time no one sees it but this was him uh from devops fame and writing the phoenix project devops handbook uh wrote just kind of shared this paper that said look this this group monitored 400 open source libraries and in just like three months 116 days they saw 282 breaking changes so more or less each day assume there's this six percent chance you're gonna have some sort of breaking change in one of your open source libraries and the reason that stuff matters and i retweeted this was that this is why your apps need to be on pipelines because your apps are probably even if your code's not changing you know your functionality isn't changing every day something is changing in your dependencies whether it's a vulnerability whether it's a breaking change it's a patch it's whatever that your apps because we now build a lot of code with a lot of dependencies it's not just encapsulated and stuff i've written i'm pulling in libraries from all over the place that means i need to be able to constantly integrate them test them and deploy them because i could be sitting here with breaking changes vulnerability for weeks or months because i only ship software once a quarter so don't put all your apps on pipelines because you're shipping 30 times a day like some web startup put your applications on a pipeline because you need to constantly update them for things like this yeah you know i've been uh uh writing up what i think is maybe the hardest section of stuff to write up

about uh you know how to run your iq better which is like a metric like i keep and that's that's difficult because uh it's one of the things where like you have to uh if you remember several episodes you know i tried to ask josh long what made his life he just wouldn't answer me so i had to like measure him a lot to draw it out from him and it's the same thing with metrics like you ask them who actually know it's important and they go like socrates socratic on you it's like oh metrics well what you know they'll never just tell you like you got to measure this you got to measure this you got to measure this and rightly so things are uh are relative but i think i think you know what uh what uh 0.1 person is looking to is uh i think of like there's sort of like two a new sort of philosophic approach to what you could call monitoring or operations or whatever and originally you have this idea of like uh the computer systems must be available and perfect and there is no further discussion and so so your your job is to keep them up and running and when it goes red you're in trouble whereas like there's just a slight new way of thinking about things which what uh what you're talking about here as a representative is is uh everything is going to be breaking all the time and what we care about is how quickly we can fix it and it's almost like uh it's sort of like a point that like mess dying or you know in in a much more abstract way uh not see how it makes right i don't know ball demand or something but essentially like what's important is to uh not be fragile and have a way of covering things which is a lot of what like uh you know we i focus on a pipeline is able to uh deliver business value or useful software sort of back to the the database point but it's also valuable for uh being able to fix things which i think is uh even now like kind of an under appreciated metric if you will uh how how quickly our how well our system is at uh fixing so

it is it is a good reminder uh or i should say proof point that basically everything is commonly decaying and we're not so much preventing entropy we're just making sure that total entropy doesn't happen uh you know before we die right yeah i think your word proof points the right thing right this is a good proof point that if you thought that app transformation was jamming stuff in containers throwing it in prod and calling it success that's not it's just no you have to also do things that can help improve your velocity which is adding test coverage adding other ways to monitor vulnerabilities or monitor dependencies that sometimes that looks like fool's gold of just hey we got to decommission some servers yeah i know but you have the same problems you had before and so i think these are nice reminders that these problems will not go away they're frankly going to increase because any modern development language now really promotes this idea of pulling in dependencies so again you don't have these sort of self-sufficient apps anymore and you're instead as you build a serverless app right you're building live pulling in libraries with thousands of lines of code so at this point we are all at the mercy of our dependencies and you should design accordingly at the mercy of our dependencies that's a good title for some of you we should suggest that to jared for his next uh tomic blog post no i think it's my autobiography title yeah exactly and then finally uh the uh like in azure there's uh they ship the i guess they ga the open service broker or they have they have that available there which is you know it's i think is emerging as a nice way to uh whether you want to call it services backing services or uh as as i i think annoyingly for the world so-called middleware you know i i like to be an old bogey but uh essentially a way to like look up uh what would you call it to use a new word other other things that you and your software rely on to get stuff done you're not necessarily responsible yeah i mean that fits on a t-shirt right there i think

that's exactly perfect very fine print yeah i mean how do you consume right it's application and data services and so the one thing you know people i think it's funny people kind of forget already like wow this open service broker is cool i wonder when it comes to cloud foundry like it started in cloud foundry like that's where that's where it came from so what's neat is that you know we took the service broker that's been in cloud foundry for the from the beginning kind of made it this more open source it was already open source but made it a more open standard now you see microsoft and other companies building these sort of brokers on that model bringing them then back to those platforms so i just thought it was a good news point good validation that this brokerage model seems like a nice way to decouple if you look at 12 factor apps one of the factors is kind of treating these services as backing services not embedded in your app but instead kind of an attachable service and so the service broker does a great job of that where my same code could just continue to be deployed the dev test and prod with no changes i'm just attaching to different broker instances and therefore connecting to different databases so really nice decoupling mechanism it's cool to see it hit this this sort of mainstream yeah yeah and i and i think i mean just in general to be some sort of prognosticator i think it's uh on on the one hand it's it's i think it's notable in the overall cloud world that we don't spend tremendous amount of time talking about see i'll do it right here middleware like like at the moment we talk a lot about how to do containers run things yeah but obviously uh you need external services to write successful software unless you're writing everything yourself which would be delightful but ridiculous so it seems like hopefully over the next couple of years there'll be a lot more discussion of here's uh here's external services or backing services or middleware how they fit into those things and in fact like i think there's there's almost like a large uh what would you call it uh glossing over of like all these existing systems that you have how do you integrate them into uh your new stuff and in fact every now and then that's probably i don't know if it's

a large part but a a used to say in corporate development a material uh amount of the discussion is about uh how do i interface with non-cloud native services make sure they don't slow me down which that'll be fun to talk about that uh more but speaking of extendable things we should talk about uh more related things now yeah yeah this is a uh a non-traditional guest ivan you should feel very special that we usually do a lot of kind of apps focus on the podcast but but you and i have gotten to be uh good buddies now and one thing we've talked about actually our last chat which is what's what kind of sparked this was you and i talked about data monoliths when we were talking and this idea of kind of data analytics monolith monoliths i should say one thing you've been helping me make sure i do is don't just say data because data could be a database data could be the literal data i mean it's very lazy sometimes just to refer to data there are companies who sell data they're right they're like uh what you call dun and brad street right and so one of the things you mentioned was data monoliths and kind of not having data monoliths much like we talk about with apps where i you know app monoliths can be beneficial in some cases but sometimes you're now coupled to a single set of gatekeepers to use them or you've got deployment challenges because it's all these big bulk deployments so what do you mean by this idea of kind of data analytic monoliths and how does some of what we're trying to do help break that up and you know thinking of the app people mindset how can they draw these parallels well i mean the the huge one of the huge trends in the in data analytics has been this idea of creating a corporate data lake and so probably that's permeated into the app conversation as well at least you probably have heard of people talking about data lakes and and we've had now it's now the data lake concept kind of started in around 2008 9 10 so we've had enough time to to kind of see it play out in a lot of cases and from observation what i see with the data lake idea is

you generally have a centrally managed team inside a large corporation who is tasked to collect all of the company data and to make it available to all the business users for for business purposes um and it it essentially they're not driven by specific business outcomes but they're they're there to provide this kind of shared service and it it turns out to to look like the the ultimate monolith in the sense that you've got non-business people driving a data architecture and a collection of storage of information um that is meant to provide value to all the different business users but then generally tends to frustrate and and and not um actually satisfy the requirements of most of the business departments i also see parallels there i know you'll keep going into the monolith side but the sort of DevOps challenge in the first place is these ops teams that are somewhat disconnected from business outcomes running these sort of shared services and kind of having to go through that but i think you also have some other monolith comparisons as well no i mean it's it's a lot of it starts from whether who's driving the project right is it is so for example if we had a marketing team let's say we're working at uh a um at widget manufacturer incorporated and the marketing team says hey we could really improve our marketing if we had good data analytics and we're going to drive a project and here's our requirements that's one point of view and the other point of view is you've got a central team in widget incorporated that says we're going to build a data architecture so the data architecture they're thinking of how to optimize for reducing data copies and and the cost of the data and how to manage the data how to track the data but they're really not coming at it from how do we do

marketing so from the from the first step you take the marketing people now have a gatekeeper which is data architects and so from the very first step now they're constantly trying to say okay what can the data architecture team give for me why not just start with the business and say hey let's build look what is the ultimate system for marketing yeah i mean that makes sense that's so when you're seeing the data lake concept fading in popularity as a whole are people trying to do things they're more departmental or more local where they're not dependent on these sort of shared data teams and data services so so i think the popularity has faded a bit but not going away but having looked at you know eight years of it and in a position where we get to see lots of different customers um there there's definitely companies that that don't build the hey we're a fortune 1000 company and here's our data lake and they build basically you know departmental specific data systems maybe using some shared infrastructure or some shared technologies but it's driven by by the business and there are other companies where i've seen build the corporate wide system and the the ones who are building the departmental systems generally tend to be from what i can see um happy and successful and the the ones building the data like either end up switching to to to spin out specific systems or um are always three to four years away in an agile project from um the ultimate delivery and i say agile is kind of as a joke right because you'll see you'll go in and you'll say we're doing this agile and in four years the data lake will be complete and it will provide this and it's always four years away i i mean this leads to the obvious question so uh how does green plum take all this what what does it do exactly and uh maybe

how does it reduce that four years well so so and and really green plum per se is a technology it doesn't sell to your guys previous discussion about if you just take a new technology and use the same mindset you're not going to fix anything right that you're talking about in the previous conversation so in the same sense you know green plum by itself is a database it's not going to solve or organizational problems and and strategy problems but the great so let's start with what is green plum it's a it's a relational database that's that's optimized for storing large amounts of data and doing analytical type workloads as opposed to rapid updates to to state so you're going to be asking questions like given given every purchase at a retail store chain in the last 10 years you know what was the average purchase volume related to um consumer goods or related to alcohol or you know so it allows you to scan through billions of records and ask questions um it's that and and that comes back to the question about the idea that there's no perfect database the world hasn't created one so every every database designer creates some sort of compromise the compromise for green plum is it's really not good at transactional processing where you're doing um highly concurrent updates right it's it's optimized for large high speed reporting type workloads and so um the reason that this comes into play as far as the data lake discussion is that um the trend was so popular that it really overwhelmed the industry to say let's basically take hdfs and hadoop and build a data lake and put everything in it and um that that trend um resulted in people in a in a kind of a approach which was to to get a central it team driving

a data project and um that doesn't mean that there aren't people who have not tried to create and and haven't successfully to some degree created corporate wide databases and enterprise data warehouses which can be done in green plum or another technology but the the technology of hadoop drove drove drove people to this approach and um they they kind of has shied away from thinking about it because it's too expensive to create a hadoop cluster for each department right and to say hey you know let's build a uh an analytical system for marketing and let's build one for finance and and let's build one for um r&d but with a database it's not too expensive you can just get 10 servers loaded up with a petabyte of data and start asking questions and it's so it's a lower overhead faster way to to take you know one petabyte of data and start storing it and asking questions and this is i mean it's a software of course so you can run this anywhere do we are most of our customers like to run this on premises on their own hardware does it require specialized hardware or do i just take anything off the shelf and run a green plum environment again it's postgres based it's about distributed mpp stuff so what's it look like yeah yeah so it's it is open source um postgres based database so it takes um postgres database and builds it um lots of them together to make one large database um as far as the the hardware the hardware really is it's what i like to tell people is that it's you need the hardware for the type of job you're doing so don't even think about green plum if if the type of your job you want to do is to take um hundreds of terabytes of data from two different tables and then correlate them you need enough hardware that can number one read hundreds of terabytes of data in a reasonable amount of time and do computation on it to do all the different comparisons and shuffling of the data over the

network so regardless of green plum if you wanted to do that type of a report you're going to need it's going to run it can run on a docker container and in a macbook air but how long will it take to do that type of a processing will take a lot longer if you don't have enough hardware to turn through the data so you need both cpu um network and disk in balanced proportions to um to basically be able to feed that data into the cpus do the processing and enough network speed for the for whatever degree you're going to be doing shuffling so what it tends to be is we we say premium economy hardware so you don't want first class okay because it's it's a way it's a waste of money um but you don't want to be sitting and coach so premium economy means you go to your favorite hardware vendor and say what do you have that's like normal hardware but it's the hottest most expensive version of normal hardware so you don't want bespoke hardware which is you know some special purpose um proprietary hardware that's expensive and because the economics aren't as good you want commodity mass-produced hardware which is going to be um give you the the right economics but then you want as high performance as possible so like today they've come out with nv ram in non-volatile ram and 100 gig networks and one terabyte ram servers so so why wouldn't you want that if you can make your your data processing run a hundred times faster in your mind and i'm maybe in your mind because i'm implying it might be delusional but what what makes green palm kind of stand out right this is a mature space this is not like analytics is not a new problem we're solving here so when you look at that yeah what stands out to you but richard the point is is that it isn't a solved it's not mature in a sense in other words there is no perfect database today at all there's no there's no like there is no database in the world that can um can hand that

can scale data end users and can scale um transactions and analytics both linearly do it with relational sql like so so it's it's a not it's non-solved problem by by humanity meaning so for example if you look at something like oracle it doesn't scale to once you get over 100 terabytes it starts slowing down um exponentially once you take green plum and start scaling it to high volumes of updates then it it doesn't it doesn't scale right and so every database has a choke point so there is no perfect database and what you want to do is look and see okay given the type of workload i'm going to run what what makes sense and then you also want to look at the platforms right so one of the huge value props of pivotal is that we sell software right um if you look at someone like amazon they're selling only services on amazon right so if if what you're looking for is an amazon service and you want to work exclusively with amazon and you want to you know be get get married to amazon for the long run then then then you can use amazon right but if you want a software product that is independent and can run where you want then then pivotal software generally is is a good approach right so the kind of open source can run on relative commodity infrastructure i think the database analytics was new to me to some extent when i joined pivotal as a i hadn't done much with that can you kind of describe what that means that what we offer there well it's a great um connection back to our excel conversation microsoft excel provides in spreadsheet analytics right you can run machine learning and or you call it artificial intelligence right because the words get used so so liberally right the artificial intelligence is a bit of a buzzword right now but ultimately

artificial intelligence is another form of machine learning machine learning is another form of analytics so you've got stuff like regression algorithm linear regression algorithms these are all available in microsoft excel right and so the idea is these are there are important mathematical and statistical functions that you can run on your data in the database you have so if you have excel and you've got this big spreadsheet why would you want to export that data to run um you know a standard deviation or or linear regression you want to run it right in place in the spreadsheet right the same way if you've got data in oracle you want to run it directly there and if you have data in green plum you want to run it directly there so the more that you're doing analytical type processing the more likely you're going to want a full suite of the most popular algorithms that you can just call as functions directly in the platform you have and so green plum has a full suite of the most popular algorithms for kind of machine learning and geospatial analytics and graph analytics and text analytics and all different types of ways to slice and dice and do math and and um algorithms on your data directly where it lives yeah that's handy again avoiding extra duplication yeah i mean why export it and especially when you have a mpp platform so if you compare it for example excel excel runs on your laptop right you can't do a linear regression in excel and distribute the work to 100 cpus but because green plum is mpp meaning massively parallel and every query you run will get subdivided by all the different computers that you have assigned to that green plum system then it's convenient to be able to um to run statistical algorithms on hundreds of terabytes of data if you can take one calculation and subdivide it automatically to all the different um servers available going going back to one thing uh you were saying i mean it is uh

it sounds like in the the database world you also suffer from i don't know what i think of as the uh the yesterday's ai problem that like once we figure out some ai thing it's everyone's used to it and it's totally like meaningless and then we're back to like ai is an impossible problem to solve it's just like at some point facial recognition was this mysterious thing and then we solve that problem right so facial recognition comes in a library called open cv right you could you could just get open cv on your laptop and and um take some snap photos from your webcam and then do facial recognition with open cv you can also take open cv and install it on your green plum cluster and have uh mpp processing of facial recognition but your point is that the word artificial intelligence gets could mean that you've got this like um intelligence that's in the in the like in some science fiction movie where it's been uploaded into the internet right in the ultimate exactly and it escaped the mainframe yeah but in in practice there's there's very specific um mathematical algorithms that have been invented that do specific jobs and those are all available and so when people say hey you know i go to let's say i go to a random customer in a random country and they say i want to do artificial intelligence they say great what do you want to do artificial intelligence like okay for what to to have artificial intelligence i'm like right to solve what once you get to do you want to have like a a cyborg computer running your business and they're like maybe once you get to a specific problem then you don't actually need some artificial intelligence you need an algorithm that can do let's say detect crop failure by looking at pictures of the farm right and that's just to your point a mundane algorithm that was invented 30 years

ago and just run it on a large amount of data yeah yeah and that that sort of bumps up against the other the other question i had before we wrap up is like how like you know you're saying you could do you could run uh you know your facial recognition over here or you could run it like in in green plum or all these things like how how do you like advise people to figure out what to run in your database versus run on your own like is there is there sort of like obvious answers to that or does it get nuanced at some point it's fair i think it's fairly obvious i mean if if your use case doesn't have any kind of requirement to have a database in it then you wouldn't use a database like if you just said i have these files of data and i want to calculate something and then i want to take those results and put it somewhere else so there's no database requirement right but once you start wanting to have storing varieties of data and start interacting with it and asking different questions now it starts sounding more like a database so the question is do you you wouldn't force the use of a database when all you want to do is do a job right let's say a batch job or a or a streaming job a very tailored specific thing but if if you want to start asking interactive questions the the beauty of the database is that you put in any arbitrary sql there can be billions of possible sqls and then it can calculate the most efficient way to answer that question without you writing the procedural code like in go or in java or whatever to to get back that result you can say start writing complex queries and say i want to select this and this and this and then group it by this and filter it by that why would you want to do that by hand but if you don't have any use case to query a database then i wouldn't put it in a database yeah and and so i mean to grossly oversimplify it's basically like uh in your application layer you're sort of like displaying

and inputting data maybe but then when it comes to like querying it and manipulating it it's probably a good default thing to like do it in your database layer rather than having your application figure all that like you know grab some big data set and sort it all for you and figure out how you want to do something it's highly likely the database will do that much better than you'll be able to code yeah i think i think exploring the data you're more likely to get benefit from a database but then once but you know a lot of times if you had a let's say an app um let's say you had a you know a taxi application right there's there's probably certain things that you want to productionize outside the database where where it's known known workflows right you said okay i want to have an api to display you know taxis within some region of me so probably that's that's a well-known enough problem that you don't need to ask arbitrary questions in a database you can maybe do it more efficiently um with some other sort of a platform where you where you create a pipeline to get that data process it push it out into some sort of in-memory cache and make it available through an api now that makes sense well then finally uh you know by nature of being open source there's a certain like visibility into the uh the team and the product management and things like that but like how like how how does how does how does the the green plum development organization work like like uh how do you go about doing doing doing the work are you on like big monolithic five-year release cycles so so it's interesting because we green plum started as a startup then was part of emc and then um became part of pivotal and when we when the green plum product became part of pivotal we we basically went through a full agile transformation ourselves i mean we had a team of 100 people in r&d and we essentially are a case study in in transforming a team from um from let's say non-agile development practice to agile development

practice and so we we use concourse um we have um we use we have weekly sprints through using pivotal tracker we use all the pivotal labs methodologies we have um pair programming um and then releases we can release within 24 hours when whenever we want to do a release so we always keep the the code shippable and um but however we do want to time releases we found that with enterprise customers there is an art to timing so they don't want to get a database update every day and they there's also when do you create api changes right so if you if it's backwards compatible that's one thing but if you if they need to change something on their code there's only a very you know certain frequency they can do that so we're shippable at any time we generally ship patches every four weeks um to keep it digestible and then um but yeah we're i think we're a great case study and i personally learned a lot on how to transform an organization into agile development piece by piece while while the car is driving that's a pretty awesome story well uh thanks for being on for people like myself who are maybe even willfully ignorant of these things what would you point them to to uh read up more what's what's a good like starting path my my starting recommendation is youtube we put a lot of energy into youtube and so we've got over a hundred different videos talking about geospatial analytics and text analytics and containerization and i would go to youtube and check that out it's easy digestible place to get some quick info yeah that makes sense for sure well great as always this has been Pivotal Conversations if you want to find our back catalog you can go to soundcloud.com slash pivotal conversation now maybe you can load all the mp3s into a database and do some kind of analysis of like what's the ratio between helpful things richard says and a name gobbly book i say

that would be a fun metric we should make a chart of that and i'll try to improve over time i just have to tell you we did a study with the talk show host and we found out that the audience started drifting when the talk show host started talking and the broadcast company ended up having to replace the host because of that analytics see that would be useful we should figure that out we can uh we can we can use some it'll it'll be like that that would be some so i guess it would be black mirror not twilight zone anymore yeah i i always think of us i was speaking of ai i always think of that one twilight zone episode where there's like a big computer and the ceo is very happy to replace all these workers and eventually the computer replaces ceo right and uh maybe maybe we should just have a really complicated query that replaces richard and i on this show right and it'll just spit up spit out some results well on on that on that optimistic note for humanity uh we'll see everyone next time bye bye

Tanzu Talk · 🍕 Get more pizza.