Welcome to the Cloud Native in 15 Minutes podcast. The goal of this podcast is simple, to explain complex but increasingly important and even strategic technologies and about the time it takes for a coffee break. We're all busy, but that's not an excuse to not understand, at least at a high level, some of the fundamental technologies that will be powering your company for the next decade. I'm Derek Harris from Pivotal, and our topic du jour is Apache Kafka. Joining me to explain is Nehar Narkedi, one of the creators of Apache Kafka at LinkedIn, and currently CTO of Confluent, a company she co-founded along with her fellow Kafka creators in order to commercialize and advance the technology. Among other things, Neha explains the evolution of Kafka, who's using it and why, and the rise of streaming data and event-driven architectures. And without further ado, here is Neha. Okay, Neha, let's start. In 30 seconds, can you explain what is Apache Kafka? Apache Kafka is an event streaming platform that essentially acts as a real-time digital nervous system for businesses. It enables organizations to connect all their applications, data sources in real time, as well as be able to react and engage to changes in what's happening across their business in real time. You know, the reason about 60% of Fortune 100 companies are using it is because they're moving from a world where data was primarily at rest, processed in batches, to a world where software and data are a significant part of their business, and that means treating data in motion versus at rest. And Kafka really helps with that foundation. You know, by building a new foundation for their data that allows you to capture really data as it appears in the real world, as streams of information. That is kind of, you know, the secret behind is the success. Okay. And a lot of people might know Kafka from its creation at LinkedIn. That's certainly how I became aware of Kafka. How has Kafka evolved as a technology since then? You know, since we created Kafka at LinkedIn, it has really evolved from a high throughput messaging
2:04queue, which was its initial claim to fame, to a full-fledged event streaming platform. And it has done that in three fundamental ways, right? The first is like providing a common foundation for a connector ecosystem so that you have a large number of connectors to all sorts of systems so you can connect those systems and move data through Kafka. The second way in which it has evolved is you have a stream processing layer. So once you have that data in Kafka, you want to process that data. And it has the ability to do that in two ways. One is through a streaming API for Java application developers. And the other is through a SQL engine called KSQL. And the third way in which it has evolved is as a core foundation for storing and processing event streams with transactional guarantees, right? And that's really a big way in its evolution since the LinkedIn days. The other way in which it has really evolved more recently as a technologies, as a serverless, elastically scalable, fully managed service in the form of Confluent Cloud. And that's, you know, LinkedIn, for example, had about more than 40 people. They still have building, managing, tuning Kafka, which is possible for maybe only 1% of companies in the world. We started Confluent to really build an event streaming platform for the other 99% by making Kafka easier to use, whether it's on-premises or in the public cloud or across hybrid environments. Okay. And if I'm an IT exec, how does Kafka tie into some of the bigger picture initiatives I might care about, right? So there are things like cloud native, microservices, even several years ago, and probably still today to some degree, big data. Like, you know, Kafka seems to touch all of those things. How do you play in those worlds? Yeah, that's a great question, Derek. In fact, you know, I somehow think that we're getting to experience a really interesting period in technology where there are several really significant paradigm shifts playing out all at the same time, right? And that's happening because companies are digitizing themselves. You mentioned a few, right? Be it adoption of the public cloud or breaking the
4:08monolith into microservices, the rise of mobile-first, real-time customer experiences, Internet of Things, or the move from batch to real-time. The most interesting thing about Kafka, and I think the reason for its steep rise in adoption is that it is the common thread that enables all of those paradigm shifts. You know, it is the new backbone for event-driven microservices. The MQs don't scale, and the databases weren't built for streaming data. Kafka is the bridge to the public cloud, right? A lot of folks use it to stream data across on-prem and cloud environments. It obviously enables real-time data transfer, and that helps while building IoT applications. But really, it is the new foundation for all data in a company, giving them a full picture of not just historical data, which they already had. But that, as well as visibility into events taking place in real-time and how that is changing the historical data. That's really, I think, the key reason why event streaming is being picked up by the Fortune 100 companies, and Kafka is becoming the foundation of their data. Okay. Can you elaborate a little bit on event streaming for listeners who might be unfamiliar? Like, how does an event streaming architecture vary, or how is it different than, you know, maybe the historical way that the companies were architecting their applications? Yeah, I think, you know, historically, there were two main data hubs for a company. You know, for transactional data, you had the databases, and for analytics, you had the data warehouse, right? So, you know, that really sufficed when technology was, you know, one of the key things that a company did. But now that every company sees themselves as a technology and a data company, you really have to take a new look at how data is treated in a company, right? So, it's no longer treated as, you know, just something that goes to rest and is queried when you need it. It actually needs to be treated as, you know, streams of information as they
6:08happen in the real world that can be studied and reacted to as that happens, right? So, if you look at databases and warehouses, they were actually not built for streams of information. They were built for data that goes to rest, and that's really the fundamental change in the event streaming architecture is the ability to store, process, move data as it arrives as streams of information. And that really requires a new foundation, you know, a continuous commit log and the ability to do stream processing. All right. And what types of specific applications are companies using Kafka to build? I think IoT, I always go to IoT, it seems like a natural application for anything involving any sort of real-time or streaming data, but I'm sure there are other common types of them that are, you know, way off from that. Yeah, I think, you know, Kafka and Confluent can be used in so many different ways and its versatility is partly why it became so popular, is it can be pretty much applied to any real-time data need. But then the thing is that every company is sort of becoming very data-oriented, so that means it's applicable in most of the things a company does. So, let me explain. From powering the new ride-sharing services to scaling connected car platforms to addressing credit card fraud, Kafka is really everywhere, right? And one of my favorite use cases is of Audi. You know, Audi is building a connected car initiative where every single Audi vehicle will have sensors that stream data. And the power of this data, you can imagine, is pretty incredible, right? For example, they want to build these new kinds of applications for all their drivers to, you know, if you have a bunch of vehicles that have the anti-lock braking system kick in at the same spot, they can work out that something's up, like a patch of black ice. So, if you're approaching that spot, they can tell you to slow down because of impending danger. They can also give predictive maintenance advice. So, you can identify that based on historical patterns, if a vehicle does X, then there might be an issue with Y and the vehicle needs to be checked.
8:09So, that's like a whole, you know, series of applications where these old-style companies are really changing their business models to use data and real-time data in particular to be more competitive. And that Kafka is really applicable in all of those cases. But there are all sorts of other examples, like if you can imagine Uber and Lyfts of the world, where these are digital native companies, right? And they're using real-time data to create new categories of businesses that simply didn't exist before. And that's the other kind of application of Kafka use cases. So, it's a versatility, continues to surprise me at times, and it's really exciting. All right. What's the difference between, or there are probably many, but what are the differences between open-source Kafka and what's available from Confluent? Yeah, you know, Kafka is transformational. In many ways, it is an open source and a really strong engine. But it really wasn't formed in times when cloud native was a primary need. It wasn't really all that easy to deploy at scale. So, our goal at Confluent is to provide more of a complete event streaming platform, right, that offers companies mission-critical reliability, but with the freedom of choice to deploy and stream across public clouds, private clouds, and hybrid environments. So, to simplify, you know, Confluent's product comes in two flavors. There's Confluent Platform, which is a software product. There's Confluent Cloud, which is a fully managed SaaS product. And the biggest difference between open-source Kafka and what's available from Confluent is really threefold. And the first is the ability to experience what cloud-native serverless Kafka means and looks like it feels via this fully managed Kafka offering, Confluent Cloud. The other is completing the Kafka experience with mission-critical security features, large ecosystem of connectors, so you can get data in and out of Kafka, having first-class clients in various languages, not just Java, a streaming SQL engine for doing
10:13stream processing, a GUI, and much more. And last but not the least, the main difference is, you know, the freedom of choice to use Kafka across hybrid and multi-cloud environments. In addition to, you know, this difference between open-source Kafka and Confluent, we actually continue to be the biggest contributor to open-source Kafka and have a deep relationship and investment in making Apache Kafka, the community, a strong and thriving place for developers who want to adopt event streaming in their respective companies. All right. Is there a point or is there, I guess, a point or a size or some criteria where you typically see Kafka users become Confluent customers? Like, what's that inflection point where? Yeah, you know, we engage with all kinds of companies at different points in their streaming adoption journey. But in the early days of Confluent, we primarily were able to recruit customers who needed help with scaling their mission-critical deployments, right? But as gradually as Confluent's product offerings became more valuable, we now help customers pretty much across all stages of their streaming adoption journey. And the value proposition really changes depending on essentially the streaming maturity of the business, right? For instance, insurance companies are beginning to take their first steps in the streaming direction. So when they become our customers, we help them with everything from education about what is event streaming to advising them on the right architecture to helping them adopt relevant parts of Confluent platform to write their first Kafka application. But on the other hand, we also have these digital native companies, right? Like, for example, Lyft adopt Confluent Cloud to steeply scale their use of Kafka without having to deal with self-managing it. So what they don't need is education on the value of streaming data. They already have that. They become Confluent customers to get access to the only cloud-native, serverless, managed Kafka offering in the market. So that said, I think as more and more companies turn into software and data companies,
12:13that's essentially what I hear customers describe themselves as we look forward to helping them across their entire event streaming journey and making this big paradigm shift called event streaming. Okay. And then finally, Neha, what are the other technologies that you see Kafka being deployed next to or alongside or as part of these event streaming architectures? Is it Kubernetes? Is it various other data tools? I'm just curious what this event streaming architecture looks like. Yeah, I think there are several tools. So there's obviously Kubernetes for entirely different reasons. There are the new age cloud-native databases. And there are machine learning tools, right? So that encompasses a pretty broad spectrum. But Kubernetes in particular is really interesting. And I think the reason for its rise is also interesting. Companies across the board, they're trying to adopt these hybrid environments, you know, just sort of hedging their bets, getting ready for faster iteration and more productivity on their development teams. And the rise of Kubernetes has been, I think, faster than most other infrastructure technologies. You know, it takes about 10 years for new things, you know, deep down in the stack to take off. But Kubernetes in the last three years has really surprised me. It really has become the new sort of foundation for, you know, computing environments, right? Sort of giving people the ability to not get locked in to any particular environment. And we see it being used across our customers more and more. And that's one of the reasons why we released Confluent Operator earlier this year and are getting ready to, you know, further deploy it, is to make it easy for stateful systems like Kafka to be deployed properly on Kubernetes, which it was not originally designed for. All right. Great. Listen, that was very informative. Thank you so much, Neha. Thanks for having me, Derek.
14:13There you have it. I hope you learned something. For more information about Kafka, you can, of course, check out the project home at kafka.apache.org. Or you can visit the Confluent site at confluent.io. For more information about our broad suite of products and services for building and managing cloud-native applications, as well as for transforming how you build software, check out pivotal.io. And if you like this podcast and want to keep up with all things digital transformation, cloud, you name it, including a curated list of the week's top stories and case studies without a lot of filler or vendor spin, check us out at pivotal.io slash intersect.