1 00:00:00,000 --> 00:00:05,200 Welcome to the Cloud Native in 15 Minutes podcast. The goal of this podcast is simple, 2 00:00:05,400 --> 00:00:09,440 to explain complex but increasingly important and even strategic technologies 3 00:00:09,440 --> 00:00:14,360 and about the time it takes for a coffee break. We're all busy, but that's not an excuse to not 4 00:00:14,360 --> 00:00:18,540 understand, at least at a high level, some of the fundamental technologies that will be powering 5 00:00:18,540 --> 00:00:23,380 your company for the next decade. I'm Derek Harris from Pivotal, and our topic du jour is 6 00:00:23,380 --> 00:00:29,680 Apache Kafka. Joining me to explain is Nehar Narkedi, one of the creators of Apache Kafka 7 00:00:29,680 --> 00:00:34,840 at LinkedIn, and currently CTO of Confluent, a company she co-founded along with her fellow 8 00:00:34,840 --> 00:00:40,060 Kafka creators in order to commercialize and advance the technology. Among other things, 9 00:00:40,420 --> 00:00:45,340 Neha explains the evolution of Kafka, who's using it and why, and the rise of streaming 10 00:00:45,340 --> 00:00:49,880 data and event-driven architectures. And without further ado, here is Neha. 11 00:00:51,980 --> 00:00:58,280 Okay, Neha, let's start. In 30 seconds, can you explain what is Apache Kafka? 12 00:00:59,360 --> 00:01:05,520 Apache Kafka is an event streaming platform that essentially acts as a real-time digital 13 00:01:05,520 --> 00:01:10,940 nervous system for businesses. It enables organizations to connect all their applications, 14 00:01:11,260 --> 00:01:18,040 data sources in real time, as well as be able to react and engage to changes in what's happening 15 00:01:18,040 --> 00:01:23,420 across their business in real time. You know, the reason about 60% of Fortune 100 companies 16 00:01:23,420 --> 00:01:27,520 are using it is because they're moving from a world where data was primarily at rest, 17 00:01:27,520 --> 00:01:32,180 processed in batches, to a world where software and data are a significant part of their business, 18 00:01:32,180 --> 00:01:37,940 and that means treating data in motion versus at rest. And Kafka really helps with that foundation. 19 00:01:38,600 --> 00:01:43,460 You know, by building a new foundation for their data that allows you to capture really data as it 20 00:01:43,460 --> 00:01:48,680 appears in the real world, as streams of information. That is kind of, you know, the secret behind 21 00:01:48,680 --> 00:01:54,880 is the success. Okay. And a lot of people might know Kafka from its creation at LinkedIn. That's 22 00:01:54,880 --> 00:01:58,760 certainly how I became aware of Kafka. How has Kafka evolved as a technology since then? 23 00:01:59,300 --> 00:02:04,880 You know, since we created Kafka at LinkedIn, it has really evolved from a high throughput messaging 24 00:02:04,880 --> 00:02:10,940 queue, which was its initial claim to fame, to a full-fledged event streaming platform. And it has done 25 00:02:10,940 --> 00:02:16,480 that in three fundamental ways, right? The first is like providing a common foundation for a connector 26 00:02:16,480 --> 00:02:21,320 ecosystem so that you have a large number of connectors to all sorts of systems so you can 27 00:02:21,320 --> 00:02:27,680 connect those systems and move data through Kafka. The second way in which it has evolved is you have a 28 00:02:27,680 --> 00:02:33,740 stream processing layer. So once you have that data in Kafka, you want to process that data. And it has 29 00:02:33,740 --> 00:02:39,840 the ability to do that in two ways. One is through a streaming API for Java application developers. And 30 00:02:39,840 --> 00:02:46,100 the other is through a SQL engine called KSQL. And the third way in which it has evolved is as a core 31 00:02:46,100 --> 00:02:53,460 foundation for storing and processing event streams with transactional guarantees, right? And that's 32 00:02:53,460 --> 00:03:00,320 really a big way in its evolution since the LinkedIn days. The other way in which it has really evolved 33 00:03:00,320 --> 00:03:06,560 more recently as a technologies, as a serverless, elastically scalable, fully managed service in the 34 00:03:06,560 --> 00:03:12,440 form of Confluent Cloud. And that's, you know, LinkedIn, for example, had about more than 40 people. 35 00:03:13,000 --> 00:03:18,700 They still have building, managing, tuning Kafka, which is possible for maybe only 1% of companies in 36 00:03:18,700 --> 00:03:24,460 the world. We started Confluent to really build an event streaming platform for the other 99% by making 37 00:03:24,460 --> 00:03:29,900 Kafka easier to use, whether it's on-premises or in the public cloud or across hybrid environments. 38 00:03:30,320 --> 00:03:37,000 Okay. And if I'm an IT exec, how does Kafka tie into some of the bigger picture initiatives I might 39 00:03:37,000 --> 00:03:41,380 care about, right? So there are things like cloud native, microservices, even several years ago, and 40 00:03:41,380 --> 00:03:46,000 probably still today to some degree, big data. Like, you know, Kafka seems to touch all of those 41 00:03:46,000 --> 00:03:47,800 things. How do you play in those worlds? 42 00:03:48,660 --> 00:03:52,440 Yeah, that's a great question, Derek. In fact, you know, I somehow think that we're getting to 43 00:03:52,440 --> 00:03:58,440 experience a really interesting period in technology where there are several really significant 44 00:03:58,440 --> 00:04:03,220 paradigm shifts playing out all at the same time, right? And that's happening because companies are 45 00:04:03,220 --> 00:04:08,820 digitizing themselves. You mentioned a few, right? Be it adoption of the public cloud or breaking the 46 00:04:08,820 --> 00:04:15,080 monolith into microservices, the rise of mobile-first, real-time customer experiences, Internet of 47 00:04:15,080 --> 00:04:21,720 Things, or the move from batch to real-time. The most interesting thing about Kafka, and I think the 48 00:04:21,720 --> 00:04:27,520 reason for its steep rise in adoption is that it is the common thread that enables all of those 49 00:04:27,520 --> 00:04:33,560 paradigm shifts. You know, it is the new backbone for event-driven microservices. The MQs don't scale, 50 00:04:33,800 --> 00:04:38,260 and the databases weren't built for streaming data. Kafka is the bridge to the public cloud, 51 00:04:38,400 --> 00:04:44,340 right? A lot of folks use it to stream data across on-prem and cloud environments. It obviously enables 52 00:04:44,340 --> 00:04:50,120 real-time data transfer, and that helps while building IoT applications. But really, it is the new 53 00:04:50,120 --> 00:04:55,440 foundation for all data in a company, giving them a full picture of not just historical data, which they 54 00:04:55,440 --> 00:05:02,180 already had. But that, as well as visibility into events taking place in real-time and how that is 55 00:05:02,180 --> 00:05:08,660 changing the historical data. That's really, I think, the key reason why event streaming is being picked 56 00:05:08,660 --> 00:05:13,460 up by the Fortune 100 companies, and Kafka is becoming the foundation of their data. 57 00:05:14,260 --> 00:05:20,640 Okay. Can you elaborate a little bit on event streaming for listeners who might be unfamiliar? Like, how does an event 58 00:05:20,640 --> 00:05:26,140 streaming architecture vary, or how is it different than, you know, maybe the historical way that the 59 00:05:26,140 --> 00:05:27,680 companies were architecting their applications? 60 00:05:28,480 --> 00:05:34,680 Yeah, I think, you know, historically, there were two main data hubs for a company. You know, for transactional 61 00:05:34,680 --> 00:05:42,680 data, you had the databases, and for analytics, you had the data warehouse, right? So, you know, that really 62 00:05:42,680 --> 00:05:50,060 sufficed when technology was, you know, one of the key things that a company did. But now that every 63 00:05:50,060 --> 00:05:56,100 company sees themselves as a technology and a data company, you really have to take a new look at how 64 00:05:56,100 --> 00:06:02,840 data is treated in a company, right? So, it's no longer treated as, you know, just something that goes to rest and 65 00:06:02,840 --> 00:06:08,580 is queried when you need it. It actually needs to be treated as, you know, streams of information as they 66 00:06:08,580 --> 00:06:14,740 happen in the real world that can be studied and reacted to as that happens, right? So, if you look at 67 00:06:14,740 --> 00:06:19,560 databases and warehouses, they were actually not built for streams of information. They were built for 68 00:06:19,560 --> 00:06:24,880 data that goes to rest, and that's really the fundamental change in the event streaming architecture 69 00:06:24,880 --> 00:06:32,580 is the ability to store, process, move data as it arrives as streams of information. And that really 70 00:06:32,580 --> 00:06:38,600 requires a new foundation, you know, a continuous commit log and the ability to do stream processing. 71 00:06:40,320 --> 00:06:45,260 All right. And what types of specific applications are companies using Kafka to build? I think 72 00:06:45,260 --> 00:06:51,060 IoT, I always go to IoT, it seems like a natural application for anything involving any sort of 73 00:06:51,060 --> 00:06:54,900 real-time or streaming data, but I'm sure there are other common types of them that are, you know, 74 00:06:54,960 --> 00:07:00,320 way off from that. Yeah, I think, you know, Kafka and Confluent can be used in so many different 75 00:07:00,320 --> 00:07:06,700 ways and its versatility is partly why it became so popular, is it can be pretty much applied to any 76 00:07:06,700 --> 00:07:11,900 real-time data need. But then the thing is that every company is sort of becoming very data-oriented, 77 00:07:12,120 --> 00:07:17,580 so that means it's applicable in most of the things a company does. So, let me explain. From powering the 78 00:07:17,580 --> 00:07:24,900 new ride-sharing services to scaling connected car platforms to addressing credit card fraud, Kafka is 79 00:07:24,900 --> 00:07:31,260 really everywhere, right? And one of my favorite use cases is of Audi. You know, Audi is building a 80 00:07:31,260 --> 00:07:37,640 connected car initiative where every single Audi vehicle will have sensors that stream data. And the 81 00:07:37,640 --> 00:07:42,040 power of this data, you can imagine, is pretty incredible, right? For example, they want to build 82 00:07:42,040 --> 00:07:48,280 these new kinds of applications for all their drivers to, you know, if you have a bunch of vehicles that have 83 00:07:48,280 --> 00:07:53,760 the anti-lock braking system kick in at the same spot, they can work out that something's up, like a patch of 84 00:07:53,760 --> 00:07:57,940 black ice. So, if you're approaching that spot, they can tell you to slow down because of impending 85 00:07:57,940 --> 00:08:04,080 danger. They can also give predictive maintenance advice. So, you can identify that based on historical 86 00:08:04,080 --> 00:08:08,960 patterns, if a vehicle does X, then there might be an issue with Y and the vehicle needs to be checked. 87 00:08:09,360 --> 00:08:15,980 So, that's like a whole, you know, series of applications where these old-style companies are 88 00:08:15,980 --> 00:08:22,560 really changing their business models to use data and real-time data in particular to be more 89 00:08:22,560 --> 00:08:28,740 competitive. And that Kafka is really applicable in all of those cases. But there are all sorts of 90 00:08:28,740 --> 00:08:33,900 other examples, like if you can imagine Uber and Lyfts of the world, where these are digital native 91 00:08:33,900 --> 00:08:40,680 companies, right? And they're using real-time data to create new categories of businesses that simply 92 00:08:40,680 --> 00:08:45,980 didn't exist before. And that's the other kind of application of Kafka use cases. So, it's a 93 00:08:45,980 --> 00:08:50,460 versatility, continues to surprise me at times, and it's really exciting. 94 00:08:51,540 --> 00:08:56,740 All right. What's the difference between, or there are probably many, but what are the differences 95 00:08:56,740 --> 00:09:00,720 between open-source Kafka and what's available from Confluent? 96 00:09:01,220 --> 00:09:07,960 Yeah, you know, Kafka is transformational. In many ways, it is an open source and a really strong 97 00:09:07,960 --> 00:09:16,740 engine. But it really wasn't formed in times when cloud native was a primary need. It wasn't really 98 00:09:16,740 --> 00:09:23,420 all that easy to deploy at scale. So, our goal at Confluent is to provide more of a complete event 99 00:09:23,420 --> 00:09:29,140 streaming platform, right, that offers companies mission-critical reliability, but with the freedom of 100 00:09:29,140 --> 00:09:34,120 choice to deploy and stream across public clouds, private clouds, and hybrid environments. So, 101 00:09:34,280 --> 00:09:38,220 to simplify, you know, Confluent's product comes in two flavors. There's Confluent Platform, 102 00:09:38,480 --> 00:09:42,140 which is a software product. There's Confluent Cloud, which is a fully managed SaaS product. 103 00:09:42,600 --> 00:09:47,340 And the biggest difference between open-source Kafka and what's available from Confluent is really 104 00:09:47,340 --> 00:09:53,480 threefold. And the first is the ability to experience what cloud-native serverless Kafka means and looks 105 00:09:53,480 --> 00:10:00,580 like it feels via this fully managed Kafka offering, Confluent Cloud. The other is completing the Kafka 106 00:10:00,580 --> 00:10:06,560 experience with mission-critical security features, large ecosystem of connectors, so you can get data in and 107 00:10:06,560 --> 00:10:13,200 out of Kafka, having first-class clients in various languages, not just Java, a streaming SQL engine for doing 108 00:10:13,200 --> 00:10:20,320 stream processing, a GUI, and much more. And last but not the least, the main difference is, you know, the freedom of choice to use Kafka 109 00:10:20,320 --> 00:10:26,920 across hybrid and multi-cloud environments. In addition to, you know, this difference between open-source Kafka and Confluent, 110 00:10:27,000 --> 00:10:32,420 we actually continue to be the biggest contributor to open-source Kafka and have a deep relationship 111 00:10:32,420 --> 00:10:38,120 and investment in making Apache Kafka, the community, a strong and thriving place for developers 112 00:10:38,120 --> 00:10:41,840 who want to adopt event streaming in their respective companies. 113 00:10:41,840 --> 00:10:50,500 All right. Is there a point or is there, I guess, a point or a size or some criteria where you typically see 114 00:10:50,500 --> 00:10:54,800 Kafka users become Confluent customers? Like, what's that inflection point where? 115 00:10:56,040 --> 00:11:01,780 Yeah, you know, we engage with all kinds of companies at different points in their streaming adoption journey. 116 00:11:01,900 --> 00:11:11,540 But in the early days of Confluent, we primarily were able to recruit customers who needed help with scaling their mission-critical deployments, right? 117 00:11:11,620 --> 00:11:20,740 But as gradually as Confluent's product offerings became more valuable, we now help customers pretty much across all stages of their streaming adoption journey. 118 00:11:21,200 --> 00:11:26,380 And the value proposition really changes depending on essentially the streaming maturity of the business, right? 119 00:11:26,380 --> 00:11:31,140 For instance, insurance companies are beginning to take their first steps in the streaming direction. 120 00:11:31,300 --> 00:11:38,820 So when they become our customers, we help them with everything from education about what is event streaming to advising them on the right architecture 121 00:11:38,820 --> 00:11:44,020 to helping them adopt relevant parts of Confluent platform to write their first Kafka application. 122 00:11:44,720 --> 00:11:48,180 But on the other hand, we also have these digital native companies, right? 123 00:11:48,180 --> 00:11:56,580 Like, for example, Lyft adopt Confluent Cloud to steeply scale their use of Kafka without having to deal with self-managing it. 124 00:11:57,080 --> 00:12:00,300 So what they don't need is education on the value of streaming data. 125 00:12:00,480 --> 00:12:02,100 They already have that. 126 00:12:02,560 --> 00:12:08,340 They become Confluent customers to get access to the only cloud-native, serverless, managed Kafka offering in the market. 127 00:12:09,000 --> 00:12:13,100 So that said, I think as more and more companies turn into software and data companies, 128 00:12:13,100 --> 00:12:20,320 that's essentially what I hear customers describe themselves as we look forward to helping them across their entire event streaming journey 129 00:12:20,320 --> 00:12:23,600 and making this big paradigm shift called event streaming. 130 00:12:24,140 --> 00:12:24,540 Okay. 131 00:12:24,680 --> 00:12:32,540 And then finally, Neha, what are the other technologies that you see Kafka being deployed next to or alongside 132 00:12:32,540 --> 00:12:35,420 or as part of these event streaming architectures? 133 00:12:35,520 --> 00:12:35,940 Is it Kubernetes? 134 00:12:36,340 --> 00:12:38,340 Is it various other data tools? 135 00:12:38,400 --> 00:12:41,460 I'm just curious what this event streaming architecture looks like. 136 00:12:42,040 --> 00:12:43,700 Yeah, I think there are several tools. 137 00:12:43,700 --> 00:12:48,440 So there's obviously Kubernetes for entirely different reasons. 138 00:12:48,800 --> 00:12:52,140 There are the new age cloud-native databases. 139 00:12:52,920 --> 00:12:54,840 And there are machine learning tools, right? 140 00:12:54,900 --> 00:12:57,500 So that encompasses a pretty broad spectrum. 141 00:12:57,780 --> 00:13:00,080 But Kubernetes in particular is really interesting. 142 00:13:00,340 --> 00:13:02,960 And I think the reason for its rise is also interesting. 143 00:13:03,540 --> 00:13:08,320 Companies across the board, they're trying to adopt these hybrid environments, 144 00:13:08,920 --> 00:13:17,320 you know, just sort of hedging their bets, getting ready for faster iteration and more productivity on their development teams. 145 00:13:17,720 --> 00:13:23,480 And the rise of Kubernetes has been, I think, faster than most other infrastructure technologies. 146 00:13:23,700 --> 00:13:28,960 You know, it takes about 10 years for new things, you know, deep down in the stack to take off. 147 00:13:29,060 --> 00:13:31,860 But Kubernetes in the last three years has really surprised me. 148 00:13:31,860 --> 00:13:39,360 It really has become the new sort of foundation for, you know, computing environments, right? 149 00:13:39,440 --> 00:13:44,500 Sort of giving people the ability to not get locked in to any particular environment. 150 00:13:44,500 --> 00:13:48,500 And we see it being used across our customers more and more. 151 00:13:48,760 --> 00:13:57,140 And that's one of the reasons why we released Confluent Operator earlier this year and are getting ready to, you know, further deploy it, 152 00:13:57,240 --> 00:14:04,900 is to make it easy for stateful systems like Kafka to be deployed properly on Kubernetes, which it was not originally designed for. 153 00:14:06,260 --> 00:14:06,980 All right. 154 00:14:07,300 --> 00:14:07,780 Great. 155 00:14:08,040 --> 00:14:10,140 Listen, that was very informative. 156 00:14:10,480 --> 00:14:11,360 Thank you so much, Neha. 157 00:14:12,000 --> 00:14:12,920 Thanks for having me, Derek. 158 00:14:13,180 --> 00:14:13,880 There you have it. 159 00:14:14,000 --> 00:14:14,880 I hope you learned something. 160 00:14:15,520 --> 00:14:21,500 For more information about Kafka, you can, of course, check out the project home at kafka.apache.org. 161 00:14:21,700 --> 00:14:25,800 Or you can visit the Confluent site at confluent.io. 162 00:14:26,280 --> 00:14:31,400 For more information about our broad suite of products and services for building and managing cloud-native applications, 163 00:14:31,920 --> 00:14:35,980 as well as for transforming how you build software, check out pivotal.io. 164 00:14:36,460 --> 00:14:41,560 And if you like this podcast and want to keep up with all things digital transformation, cloud, you name it, 165 00:14:41,860 --> 00:14:47,280 including a curated list of the week's top stories and case studies without a lot of filler or vendor spin, 166 00:14:47,460 --> 00:14:50,300 check us out at pivotal.io slash intersect.