Welcome to the Cloud Native in 15 Minutes podcast. If you haven't listened before, the goal of the podcast is simple, to explain some complex but increasingly important and even strategic technologies in about the time it takes for a coffee break or a drive to the office or a walk or whatever it is you do in 15 to 20 minutes. We're all busy, but that's not an excuse to not understand, at least at a high level, some of the fundamental technologies that will be powering your company for the next decade. So I am Derek Harris from Pivotal, and the topic today is SQL, specifically new slash distributed slash next-gen SQL, the type of databases that are focused on scale, simplicity, performance, and quite often geographic distribution. Joining me to explain this new class of SQL databases is Peter Mattis, the co-founder and CTO of Cockroach Labs, the creators of CockroachDB. And I should first issue a disclaimer where we ran into some audio issues recording and then attempting to re-record this episode. So please bear with us. It's not unlistenable. It's just not ideal. I posted a cleaned up transcript of this episode at cloudnative.simplecast.com and will include it in the accompanying blog post as well if you prefer to take in the content that way. As for the content itself, Peter explains why you're hearing so much right now about, you know, these newer next-gen or distributed SQL databases, a group that includes CockroachDB, but also offerings from other startups such as YugoBite and all three major cloud providers. He also dives into the connection between these newer SQL databases and the cloud native world, specifically Kubernetes, and why large enterprises should feel comfortable trusting open source startups for their mission-critical applications. So stick around and listen to Peter Mattis of CockroachDB now. Okay, so let's get into it. Peter, can you explain in about 30 seconds what Cockroach Labs and or CockroachDB are and kind of, you know, just this idea of distributed SQL in general?
2:01Yeah, the idea behind distributed SQL came out because we noticed there's like kind of a gap in the landscape. There's these NoSQL databases, which were coming for a company for a company, and they were promising easy, horizontal scalability. And yet they tossed out so many, you know, easy-to-use functionality that developers like, like indexes and transactions. And on the other side of the spectrum, there are traditional SQL databases that provide indexes and transactions and all sorts of other goodness, but they didn't have that horizontal scalability. So we wanted to build a system that had the best of both worlds. Easy, horizontal scalability with all the SQL goodness that makes developer lives nice. And what's happening today, I guess, like business-wise, technology-wise, et cetera, that makes, because there's a bunch of, you know, let's call them new SQL or, you know, variants on SQL, right? What's happening right now that makes these new variants on SQL databases so valuable? Well, the thing that's, I would look to not so much as valuable as possible. Why are these all coming out kind of near the same period of time? It's like, why didn't, you know, multiple people independently come up with the idea of the atomic bomb and groups come up with, they kind of come with these breakthroughs? Well, there's a kind of technological change that occurred over the past 40 years since databases have been in existence. And that is, you know, computers have gotten faster. We all know about Moore's Law, but networks have even outpaced the improvements in processors. And now it's, you know, I can read data from a remote machine, especially one that's nearby in a data center, faster than I can read it from disk. That's pretty incredible. Not a lot of people realize it outside the industry. Inside the industry, it's well known. And then the other big change is that we actually have a much better understanding of how to do, you know, distributed transactions and distributed consensus. All this stuff has come out in the past 20 years about Paxos and Raft, which are distributed consensus protocols. And then there's distributed transaction protocols that exist on top of that. So I think there's like a realization by a whole bunch of different people
4:02that you can put these things together and make a distributed SQL system. And what we're seeing is, you know, various different, you know, kind of ways to piece together these components into a working system. All right. Are there specific application types that benefit from that? I mean, I'm just curious, like, why I'm interested in using this. Yeah. So if you were trying to use like a NoSQL system, like any of the applications that were being built on top of NoSQL systems, you know, the systems I'm thinking here, like inside Google would have been Bigtable and then outside, say, you know, systems like Cassandra and HBase. You can build those on top of something like CockroachDB as well. And the workloads those work well for are ones that, you know, you're doing a mixture of reads and writes. CockroachDB provides better transaction semantics for those workloads. So some things that might have been awkward to do in Cassandra are a little bit more natural to do in CockroachDB or significantly more natural to do in CockroachDB. Then one of the things that a distributed SQL, in particular CockroachDB's distributed SQL unlocks, is the ability to do multi-region clusters and have geo-distributed replication. With the emergence of applications that have global concerns, global concerns in the sense of, you know, regulatory compliance concerns like GDPR or just users that are spread throughout the world, people who have a database that, you know, scales and can adapt to where those users are is an important bit of new functionality that we provide. Okay. And I mean, you touched on this, but I wanted to ask specifically, is something like CockroachDB or, you know, pick something similar on the market, like a replacement for an existing Postgres or MySQL plus NoSQL databases, or are they targeting really, like you mentioned, kind of different workloads? Oh, they're absolutely a replacement. I mean, if you were to use Postgres or MySQL directly, I mean, these are functional, you know, kind of battle-tested databases, and yet setting up the replication is a headache.
6:03Dealing with the failover is a headache. And dealing with that, the scalability, you know, once you get to a certain size limit for Postgres or MySQL instance, you know, you actually end up having to do sharding. And that sharding is this huge burden on application developers. It requires them to put logic in their application to worry about how I'm spreading my data across multiple, you know, Postgres or MySQL instances. So any of those workloads that you were putting on, you know, Postgres or MySQL before, you can put on CockroachDB, and we just handle that sharding for you. We handle all the replication just out of the box. The availability story is just much, much smoother. Okay. And is this a matter of, I mean, do the users typically come at a certain size or scale? I would imagine, right, if I'm a two-person shop, I might or might not need, you know, that level of replication, distribution, or is, I mean, just, yeah. So where is, like, the usual user, right? What's that profile look like? You know, we see a kind of mix. You know, we see people who have a relatively small amount of data and just want to make sure that, you know, that, you know, the high availability story that you provide is much, much cleaner. And we also see people in shops that are, you know, talking about, you know, very huge workloads. I mean, like, if you're just a two-person shop, one of the things that, you know, the industry as a whole is moving towards is database as a service. You're providing the database as a service. You know, if you're a two-person shop, you don't want to be running Postgres or MySQL yourself. You're probably wanting to run, you know, RDS or whatever the equivalent is on Google. And we're also offering Cockroach as a service via a, you know, managed service offering. That's currently in beta and we expect to launch it into wider availability this fall. All right. And if I decide to adopt a distributed SQL database, what do I need to do differently in terms of managing it, in terms of writing to it? I mean, I assume it's not a one-to-one mapping in terms of skills, but, you know, maybe you're going to tell me differently. Yeah. Well, there is a lot of overlap and that's one of the reasons
8:03that we actually adopted a SQL as the interface language for CockroachDB is single as lingua franca for data manipulation industry. Now, there are just hundreds of thousands, millions of developers who know SQL and know the intricacies of SQL. And most of those just map perfectly onto CockroachDB. The thing you do have to be aware of is when you're going to need to multi-region deployments that you are having to be concerned about the inner-region latencies. We don't violate the laws of physics. They're, you know, your data centers are, you know, on different sides of the continent. There's going to be latencies between them. But when you know about those latencies, you can take them into account in your schema design. You know, it's essentially equivalent to the realization that, oh, I need to know about the concept of index inside a database and what impact that's going to have on performance. There's a similar bit of concern with regards to, like, the geo-distribution of your data. Need to take into consideration, but it's something people can wrap their heads around. All right. And why not, you know, just use, I mean, like there are so many cloud providers out there offering, you know, other sorts of these distributed databases, right? Why not just use, you know, Amazon Aurora or Cosmos or Google Spanner or something? What's the benefit of actually, you know, deploying a, whatever you want to call it, you know, deploying software, I guess, right? Or deploying a specific database. Yeah. Well, I mean, there's multiple, you know, kind of answers to this. I mean, one is we're offering capabilities there that you don't get from Spanner or from Aurora yet in our geo-distribution capabilities. The other big answer that affects a ton of customers is that there's that vendor lock-in. If you're going with Aurora or you're going with Spanner, you're locked into those vendors. And that can be a significant hardship, especially if you're trying to get into, you know, these big cloud providers, they're providing data centers all over the place, but sometimes they have a gap in their coverage and that gap can affect your users. But I think it's also
10:04that kind of cloud lock-in that scares a lot of people. And, you know, this is becoming more and more of a concern. Amazon is just, you know, the huge elephant in this room. It's gobbling up kind of mindshare. You know, are they benevolent or not? And, you know, it's maybe a little bit too early to tell, but even if they are benevolent, they can just accidentally turn and, you know, stumble over, you know, your company and crush it. And, you know, I think a lot of people are starting to become concerned about that. All right. It's funny. So it seems like there's almost a, especially in the database world to some degree and the distributed and kind of next-gen database world, right? There are these big, big cloud providers and then there are a lot of startups, right? So why would, I mean, locking aside, why would I trust a startup if I'm a CIO or, you know, someone with a decision-making, you know, and budget? Like, why would I decide, why would I, why would I trust a startup for to do something mission-critical instead of, say, you know, one of these big cloud providers? Yeah, no, it's an excellent question. I mean, one of the reasons we were providing the source code and we went open source, now we have a slightly, you know, kind of modified where our license isn't pure open source, but it still provides a lot of the same benefits is that, you know, the concern is, oh, if we just suddenly fold up shop, what happens to us? And this at least gives some, you know, modicum of, you know, people still have access to the source and a community can form around it to support it. I mean, that's kind of like our kind of fail-safe of, you know, concern. The other thing is, you know, like Postgres, MySQL, they're heavily invested by these big cloud providers and yet, you know, they're still open source themselves. They're, you know, to some degree, it's not like these are like the flagship database projects that people are putting their data on. They're just developed in the same way that the cockroaches. You know, we test the, you know, put significant effort into testing for correctness and stability and performance for our system. It's essentially like all that same development practices that they're doing, we do as well.
12:04All right. And then, I mean, speaking of the cloud, I guess, how would, how do some of these new SQL databases relate to other cloud native technologies and practices, right? You might look at microservices or Kubernetes or Kafka, serverless, and functions. Like, is there, is there a kind of an inherent connection or integration with some of these other new practices? Yeah, certainly Kubernetes is the one, you know, there's this huge wave occurring in the cloud, which is everybody's moving to Kubernetes and putting their microservices on top of Kubernetes. And for the most part, that works well. Kubernetes provides a great substrate for running stateless applications. But then we have the stateful applications, and these are the ones storing data in databases. And then you have to wonder, where is the database itself running? Cockroach actually runs really nicely on top of Kubernetes. And that is our number one distribution mechanism. The number one mechanism by which people run CockroachDB is on top of Kubernetes. And then they'll have, you know, Cockroach running Kubernetes, their application running on Kubernetes, you get all the goodness from the Kubernetes scheduling. One of the things that these kind of, this new wave of distributed, you know, SQL databases is doing is because we have the replication and whatnot just kind of built in and baked into the product that it handles that elastic scale-up that Kubernetes offers and it handles the fail-up that Kubernetes offers where you don't get the same behavior from something like trying to run MySQL or Postgres on top of Kubernetes. Okay. And do you see a common or at least a relatively common kind of data architecture as you're looking at customers and what they're running? The idea of Kubernetes as a substrate for running stuff is intriguing, right? So I'm curious if there's kind of a standard architecture of Cockroach or SQL Plus whatever that you've seen pretty commonly. Yeah. You know, I can't say like we see so much above the stack that's common. There seems to be a little bit more froth there but certainly, you know, Kubernetes as a substrate, this has actually just completely changed
14:04over the past four years. When we were getting started with Cockroach Labs, you know, I think Kubernetes was announced just right around that time or just slightly afterwards and it's like, oh, let's see how this goes and now it's just every conversation and every customer is asking about this and this goes from, you know, the small startups all the way up to, you know, the big fortune 500 companies that are using us and experimenting with us. They all have some kind of Kubernetes mandate it feels like. So that's really the common thread we see going on there. But as far as like, you know, like kind of the other parts of the stack, you know, people using us from JavaScript, from Java, from Go, from Python, you know, it's just, it's quite a bit more diffuse there. All right. And then, I mean, you touched on the open source nature of Cockroach and a lot of other, you know, of, you know, maybe Cockroach alternatives are open source as well. Like, what's the reason behind the open source, you know, behind the open source? Because it seems like most new databases, frankly, you know, or at least many are open source. And then how should enterprises think about using, you know, the free versus the paid versions of those products? Yeah. Well, there's kind of two questions there. So, I mean, the first question is, you know, what's the motivation behind, you know, choosing, using open source for these products? And part of that is, you know, there's been some, you know, problematic history behind closed source databases provided by Starps. Foundation DB was the one that occurred just right around the time, you know, four years ago that Cockroach Labs was getting started where they were closed source, kind of not quite new SQL. They weren't quite as advanced as CockroachDB in some ways. They were more of a key value store type no SQL, but they were closed source and, you know, they looked like they had a really nice architecture and everything and then they got bought by Apple and got shut down. And, you know, that was kind of, you know, we were paying attention at the time being like, ah, you know, customers are going to have a hard time adopting us if, you know, the rug can get pulled out at any time. So, that was one of our motivations for going open source is to, you know, provide this guarantee.
16:04It's like, you know, no matter what else happens to us, we have bought, whatever, we go under, you know, that this is going to be there and you have that fail safe there. Another aspect is just the marketing, the go-to-market of using open source. There's a kind of heavyweight go-to-market asking people like, hey, try my closed source database software and, you know, you usually have a kind of heavy sales effort involved there where, you know, it's like, you want to get the access to the binary for trial purposes, three-day trial and whatnot. What open source does is it allows you to kind of go in the back door. Developers can just download your software, start trying it, kicking the tires and whatnot. They can even peek under the hood to see if the claims you're making on your website are valid. You know, you hear someone says, oh, this is absolutely reliable and then you actually peek under the hood and you see what their coding practices are and see what their testing practices are and, you know, kind of, you know, your eyes can open wide and be like, oh, no way would I trust that. And what we do is all in the open. People can, you know, follow along. Not sure if they do, but that's part of the, you know, the rationale behind it. Kind of keeps us honest to have that openness baked into what we do. Okay. And then the second part of the question, Peter, was how should enterprises think about using or I guess any users think about using the free versus paid version of an open source product? Yeah. So, I mean, one of the main reasons that people use our paid version is we actually put features into that enterprise version that you want to have if you're running a production. And we kind of try to draw the line between our free version and our paid version based on things that, you know, a small startup might not necessarily need when they're just getting started, but you almost certainly want when you're getting to any sort of scale. Most obvious example of that is Backup and Restore is on our paid version. And, you know, like any big enterprise that's using a database product is probably going to want Backup and Restore. But above and beyond that, you know, there's usually, you know, support contracts that come along with our enterprise version. And that's also kind of a critical need
18:04for any enterprise. If you're putting, you know, we want people to put their crown jewels with their data into CockroachDB, it's kind of almost a fiduciary duty to have a support contract at that point with the company providing that software. And then, and then finally, Peter, I mean, what does your ideal future look like with regard to SQL and database technology in general, right? I mean, that's in terms of capabilities, I guess, but also in terms of how you would like to see organizations and users, you know, architecting and building new applications to take advantage of what's coming down the pike. Yeah, I mean, the thing I'm excited about is the, you know, this evolving landscape of applications that can work in a global manner and have geodistributed applications that can take advantage of a geodistributed database. And there's going to be enhancements to the kind of the SQL abstractions to make this easier to do. And I'd say we're still kind in the early days of this. CockroachDB provides a lot of mechanism for doing geodistributed applications. And yet those mechanisms are a little bit, you know, rough around the edges. So, I mean, the area I'm excited to see innovation in is improving those SQL abstractions, improving how, you know, that functionality works. So it's just crystal clear and easy for developers to use. And then also seeing this evolution of the, you know, kind of a global app tier come into play. You can, so easy to get a VM running in a data center, but actually getting an application running is quite a bit harder than trying to get a geodistributed application running is even harder still. And all that stuff is going to be addressed over time. We're seeing progress in these areas. You know, before it was starting a VM, now it's like usually you get a Kubernetes cluster running, but getting a multi-region Kubernetes cluster is still hard. I think these things are all going to be addressed in the coming years. And we're going to get to this point where, you know, if I want to, you know, spin up a global application, I'll be able to do that in minutes and deploy it in minutes. You know, that's still, you know, kind of a far off pie in the sky, but I think we'll get there. All right. Great. And with that, we are out of time. Thanks a lot, Peter.
20:05All right. Well, thank you, Derek. There you have it. I hope it was 20 minutes well spent. For more information about Cockroach Labs, the company, and CockroachDB, the product, you can visit CockroachLabs.com or you can dive a little deeper into the project at GitHub.com slash CockroachDB. For more information on our broad suite of products and services for building and managing cloud-native applications, as well as transforming how you build software, check out Pivotal.io. And if you like this podcast and want to keep up with all things quote-unquote digital transformation, including some insightful posts from experts, some guides and explainers on cloud-native technology, and even highlights from these podcasts, you can visit Pivotal.io slash intersect as in the intersection of business and technology. That's Pivotal.io slash intersect. Thanks for listening.