Welcome to a special episode of the Cloud Native in 15 Minutes podcast, where over the extended holiday season, we'll be sharing some of the highlights from the show's first six months. I'm Derek Harris from Pivotal, and if you follow the show on the Intersect site, Pivotal.io slash intersect, you know I put together a blog post for every episode where I pull some of my favorite excerpts from the interview. What you might not realize, however, is that those quotes are quite often condensed and edited for clarity. And so what I want to do here is provide those same quotes with a little more context and more flavor, more audio flavor of the actual conversation. This first best of episode features highlights regarding cloud native application architectures and development and specific technologies ranging from Kafka to Kubernetes to SQL, and it features some of the co-founders or creators of Kubernetes, Kafka, MongoDB, CockroachDB, and Snyk, as well as Pivotal's own cloud native expert, Cornelia Davis. And I should note that I omitted some highlights in the name of keeping this episode at least in the ballpark of 15 minutes, but the TLDR version is that enterprises are very interested in running seemingly everything on top of Kubernetes. So everything we talked about here, people want to run on top of Kubernetes. And one quick reminder before we get into the highlights, if you enjoy the podcast, please share it with your friends or family or more likely colleagues and rate it in the app or platform of your choice. Thank you. Up first, we have Neha Narcady, the co-founder and CTO of Confluent and one of the creators of Apache Kafka during her tenure at LinkedIn, explaining how Kafka relates to any number of other cloud native technologies, including microservices, IoT, and even the public cloud. Yeah, that's a great question, Derek. In fact, you know, I somehow think that we're getting to experience a really interesting period in technology where there are several really significant paradigm shifts playing out all at the same time, right? And that's happening because companies are digitizing themselves. You mentioned a few, right? Be it adoption of the public cloud or breaking the monolith into microservices, the rise of
2:01mobile first real-time customer experiences, internet of things, or the move from batch to real-time. The most interesting thing about Kafka, and I think the reason for its steep rise in adoption, is that it is the common thread that enables all of those paradigm shifts. You know, it is the new backbone for event-driven microservices. The MQs don't scale, and the databases weren't built for streaming data. Kafka is the bridge to the public cloud, right? A lot of folks use it to stream data across on-prem and cloud environments. It obviously enables real-time data transfer, and that helps while building IoT applications. But really, it is the new foundation for all data in a company, giving them a full picture of not just historical data, which they already had, but that as well as visibility into events taking place in real-time and how that is changing the historical data. That's really, I think, the key reason why event streaming is being picked up by the Fortune 100 companies, and Kafka is becoming the foundation of their data. And here is Neha explaining event streaming and why that is becoming such a popular architecture. Yeah, I think, you know, historically, there were two main data hubs for a company. For transactional data, you had the databases, and for analytics, you had the data warehouse, right? So, you know, that really sufficed when technology was, you know, one of the key things that a company did. But now that every company sees themselves as a technology and a data company, you really have to take a new look at how data is treated in a company, right? So it's no longer treated as, you know, just something that goes to rest and is queried when you need it. It actually needs to be treated as, you know, streams of information as they happen in the real world that can be studied and reacted to as that happens, right? So if you look at databases and warehouses, they were actually not built for streams of information. They were built for data that goes to rest.
4:02And that's really the fundamental change in the event streaming architecture is the ability to store, process, move data as it arrives as streams of information. And that really requires a new foundation, you know, a continuous commit log and the ability to do stream processing. Up next, we have Joe Beda, one of the creators of Kubernetes during his time at Google and now a principal engineer at VMware, explaining in his mind, the difference between cloud computing and cloud native computing. I think it's helpful here to tease apart like cloud versus cloud native. In my mind, cloud is running on somebody else's infrastructure, or at least something that somebody else is managing for you in sort of private versus public cloud. But as a user, you're like, I'm not worrying about the nitty gritty details. Generally, it's about API driven, self-service in that you don't have to talk to anybody. I like to call public cloud, not talking to a sales guy as a service. And then it's a last thing, meaning that, you know, there's a lot of room to essentially bring new techniques because you can have infrastructure on demand. So that's cloud. Cloud native in my mind then is what are the techniques? What is the structure? What is the set of tools that you use so that you can take the best advantage of? How do you actually take those new capabilities that cloud offers and, you know, apply those to your organization and then take that to the fullest extent that makes sense over time? Right. So that, I think, is cloud native. And so you see a lot of big companies, you know, I'm using air quotes here, move to cloud and they'll spend millions of dollars and like months and months or years of time. And then they'll get the whole thing done and they'll be like, how come we didn't magically turn into Netflix? Right. Which is like the poster child for Amazon. Right. And the reason being is that they are physically running on cloud, but they haven't actually changed their practices, changed the way they do stuff. So they brought all their old sort of processes, their old ways of doing things, and they're just
6:02now running on somebody else's computers. And here's Joe again, this time responding to my statement that organizations who aren't familiar with Kubernetes and cloud native development risk cutting themselves on sharp edges using pure open source Kubernetes. Well, I think that's true. But also the flexibility of Kubernetes means that they can, they don't have to be purists. There's room to compromise as they look to adapt their old stuff into their new stuff. Right. Like, you know, we have a very large financial services company and they had a sort of home built big data ish sort of batch processing system. They're looking to move that stuff to Kubernetes, but there's a lot of roadblocks if they were going to go to a sort of purist type of route to this. But the flexibility of Kubernetes means that they can, they can start getting used to it. They can start moving forward without having to necessarily, you know, rip and replace everything that they've been doing. And next we have Elliot Horowitz, the co-founder and CTO of MongoDB explaining why it is that organizations or developers choose MongoDB and frankly choose non-relational databases in general. I think, you know, I think it comes down to a few things. One is there's a lot of places where, you know, the data model for an application just doesn't fit into a relational data model. And then MongoDB is sort of a no-brainer. There are cases where none of the relational databases scale to the kinds of volumes people need. I mean, that's a great use case for MongoDB. There are cases where certain features like being able to put data spread around the world in a single cluster are a critical feature, whether, you know, either for compliance reasons or latency reasons. And you want to keep data in different countries and close to users. And that's something where only something like MongoDB could really satisfy it. It's not really people who are saying, oh, I've got an Oracle database. Let me just swap it out with Mongo just for fun. Right? It's a lot of work. And so it really comes down to some business problem, some real problem saying, I've got
8:02a problem here. My answer is doing a re-architecture inside of Oracle and figuring out a totally different way to use Oracle or moving to MongoDB and making my life, you know, my developers' lives a lot easier. And here's Elliot explaining why MongoDB tends to be a popular choice as a data store for microservices, as well as IoT and kind of novel data use cases where MongoDB is picking up steam. MongoDB tends to be used in a lot of microservices use case because it fits very well to the paradigm of, you know, hey, I've got this application, this application or this service is going to use this kind of a data model and, you know, have a very tight mapping. It's very easy to have lots of, you know, MongoDB instances that can scale with a service and therefore it works well. A lot of companies we see take these big monolithic applications built on Oracle and their way off of this monolithic environment is to, you know, peel off services one at a time and as they do that, they tend to often move them to MongoDB as well, right? So take this, you know, a hundred different applications using the same database, start pulling off services each one. As they pull off a service, they also move the data to MongoDB. That's pretty common. We're used in a lot of big data cases and a lot of IoT cases, you know, where people are putting in a ton of sensor data, streaming data, whether it's, you know, sensors or financial data. So it works well in that paradigm largely because it scales very effectively and because of the data model, again, because you can put arbitrary data in there. So as your sensors change, as the world around you changes, the database doesn't have to change with it. It just sort of naturally evolves with you. And up next, we have one of my colleagues, Cornelia Davis, vice president of technology at Pivotal, explaining the evolution of platform as a service or PaaS from something focused solely on the developer use case to something that thinks about the whole application lifecycle. So from infrastructure to security, all the way up to, again, the application. What I think we've learned, and so what has really changed, is that our understanding of
10:02the potential value that something like a platform as a service can bring goes so far beyond making Cornelia's life easier, making the developer's life easier, and takes you all the way to the entire lifecycle of the application. So it's not just about development, but it's actually about deployment to production, operating in production, resilience in production, and even the developer workflow is considered a production activity because as soon as that environment goes down, then you can't be shipping features anymore. And so I think that understanding that it's not just about making the developer's life easier has led to a maturity in PaaS that now allows us to focus on things like security and operational effectiveness and resilience, which when I first started studying this seven plus years ago, nobody was talking about it. And here's Cornelia explaining some of the differences between PaaS and what some people referred to as Kaas, C-A-A-S, or containers as a service, specifically Kaas as delivered via Kubernetes. Kubernetes is crazy cool technology. It's a set of primitives that you can just build so many awesome things with, but that's the key, is that you can build awesome things with it. The main thing that I think I would love your listeners to understand is that Kaas is essentially an infrastructure dial tone. So when you are dealing with Kubernetes, you are dealing with things like pods, and you are dealing with persistent volumes and persistent volume claims, and load balancers, they call them services, but a load balancer is a type of a service. So you're still largely dealing with infrastructure abstractions.
12:02Yes, it's an up-leveling from compute storage and network. It isn't just those broad things. It's not a machine in the same way that we've always thought of machines. It used to be a physical machine. Now it's a virtual machine, and it has every single thing that the physical machines had. We're starting to break those things up into an abstraction layer where we can compose them in interesting ways, but it's still fundamentally an infrastructure abstraction. So paths is all about bringing that abstraction up to a level that lets the infrastructure details fall by the wayside, that lets you apply controls to those, but then they aren't first-class concerns that's already built in. And here we have Guy Pujarni, co-founder of Sneak and previously CTO at Akamai, talking to us about why and how open source and other shifts in technology have changed the name of the game for securing applications. Open source is a component of DevOps, if you will, and a whole stack of changes that really transformed the way software is developed, developed faster. Software just sort of gets written and gets deployed. The path from writing a line of code to deploying it is much, much faster, and it is continuous. It doesn't stop. It's from CICD to a myriad of other technologies, including open source components and including containers, it just became continuous. And that really kind of rocks the boat for security because security's natural motion has been to work through gates, right? It's you stop here and you will audit. And so as software becomes fluid, right, becomes continuous, the opportunity to do those stops and to audit goes away. And that forces, that changes the world of security to need to embed itself more into the pipelines, into that continuous process, as that's really its opportunity. And here's Guy again with a very insightful comment on why open source is not more or less secure
14:07than anything else. It's just differently secure than some perhaps more traditional approaches. Definitely open source is not less secure nor more secure than closed source code. And it's not, you know, nor is, you know, a DevOps methodology, right, or a continuous pipeline more or less secure, you know, than a different approach. They just have different properties. Specifically, I do come back to people. So when you think about open source, for instance, it does two things. One is prevalence. So because open source is there, because you don't want to reinvent the wheel, then you go off, you have a piece of code that is very, very useful, and everybody embraces it. So it's not that that piece of code, OpenSSL or Bash or, you know, the Docker engine or whatever it is, is more or less secure. It's just that it's amazingly prevalent, you know, just far more so than any commercial software. So when a vulnerability does get found in one of those components, it has a whole world of victims available to it. So its implication on the entire internet ecosystem could be seismic, could be far more substantial than commercial software. So in that sense, open source is not more or less secure, but its security is more important. The likelihood of an attacker exploiting a vulnerability in an open source component is just higher. And you can kind of make the argument that a cloud application versus one that is just sort of a hidden in the data center, if it is more exposed to the world, it provides more value, but it also increases its attack surface. So that's one aspect. But the other and maybe very people-oriented aspect of open source is one of ownership, right? And it has to do with the fact that an organization that uses open source is getting a piece of value for free, but they're downloading it from the internet and their use of it is at their responsibility. So organizations need to build a new muscle, right? They need to understand how do they use this software that they did not write and take ownership of securing it,
16:09even though it's not their developers that wrote it. And that's, you know, I think that's a very substantial change to how companies think about managing their software. And finally, we have Peter Mattis, the co-founder and CTO of CockroachDB. He starts off with explaining why we're seeing so much new slash distributed slash next generation SQL today, as opposed to seeing it maybe a decade ago. I would look to not so much as valuable as possible. Why are these all coming out kind of near the same period of time? It's like, why didn't, you know, multiple people independently come up with the idea of the atomic bomb? You know, groups come up with, they kind of come with these breakthroughs. Well, there's a kind of technological change that occurred over the past 40 years since databases have been in existence. And that is, you know, computers have gotten faster. We all know about Moore's law, but networks have even outpaced the improvements in processors. And now it's, you know, I can read data from a remote machine, especially one that's nearby in a data center, faster than I can read it from disk. That's pretty incredible. A lot of people realize it outside the industry. Inside the industry, it's well known. And then the other big change is that we actually have a much better understanding of how to do, you know, distributed transactions and distributed consensus. All this stuff has come out in the past 20 years about Paxos and Raft, which are distributed consensus protocols. And then there's distributed transaction protocols to exist on top of that. So I think there's like a realization by a whole bunch of different people that you can put these things together and make a distributed SQL system. And what we're seeing is, you know, various different, you know, kind of ways to piece together these components into a working system. Here is Peter giving his thoughts on what he's excited about for the future of databases, primarily the benefits of databases and applications designed for handling geographically distributed workloads. Yeah, I mean, the thing I'm excited about is the, you know, this evolving landscape of applications that can work in a global manner and have geodistributed applications that can take advantage of a geodistributed database.
18:13And there's going to be enhancements to the kind of SQL abstractions to make this easier to do. And I'd say we're still kind of in the early days of this. CockroachDB provides a lot of mechanism for doing geodistributed applications. And yet those mechanisms are a little bit, you know, rough around the edges. So, I mean, the area I'm excited to see innovation in is improving those SQL abstractions, improving how, you know, that functionality works. So it's just crystal clear and easy for developers to use. And then also seeing this evolution of the, you know, kind of a global app tier coming to play. You can, it's so easy to get a VM running in a data center, but actually getting an application running is quite a bit harder. But then trying to get a geodistributed application running is even harder still. And all that stuff is going to be addressed over time. We're seeing progress in these areas. You know, before it was starting a VM, now it's like usually you get a Kubernetes cluster running, but getting a multi-region Kubernetes cluster is still hard. I think these things are all going to be addressed in the coming years. And we're going to get to this point where, you know, if I want to, you know, spin up a global application, I'll be able to do that in minutes and deploy it in minutes. You know, that's still, you know, kind of a far off pie in the sky, but I think we'll get there. And there you have it. I hope you enjoyed listening to some of these highlights as much as I enjoyed re-listening to them. And if you want more about digital transformation and cloud native architectures and the whole gamut of things discussed in this episode, you can obviously check us out at Pivotal.io, or you can check out Intersect, the home of this podcast, at Pivotal.io slash Intersect. I'm also running the site on Twitter at Intersect IT.