1 00:00:00,000 --> 00:00:05,620 Welcome to the Cloud Native in 15 Minutes podcast. I'm your host, Derek Harris from VMware. 2 00:00:06,680 --> 00:00:12,420 And I'll skip the long intro I used to give because, well, this podcast is not 15 minutes, 3 00:00:12,440 --> 00:00:16,920 it's closer to 30 minutes. But if you listen to it at 2x speed, you'll about hit that 15-minute 4 00:00:16,920 --> 00:00:23,800 mark. So joining me, my guest today is James Urquhart of VMware's Strategic Executive Advisory 5 00:00:23,800 --> 00:00:29,800 Office. And James, I guess he's a Strategic Executive Advisor. And James talks a lot, 6 00:00:29,840 --> 00:00:32,900 he's very knowledgeable about distributed systems. And in this particular podcast, 7 00:00:32,900 --> 00:00:38,380 we talk a lot about event-driven integration and event-driven architecture and why that is where 8 00:00:38,380 --> 00:00:42,140 a lot of applications are heading and why it's valuable in a world where you want real-time 9 00:00:42,140 --> 00:00:48,920 insights and being able to act in real time on events that are happening and the tools and the 10 00:00:48,920 --> 00:00:53,360 processes and the organizational changes that need to happen to make that happen. So I think it's a 11 00:00:53,360 --> 00:01:00,640 really, really insightful interview. James is a really insightful person on this topic and other 12 00:01:00,640 --> 00:01:05,240 topics. And he's writing a book about event-driven integration that he references. So that should 13 00:01:05,240 --> 00:01:10,340 be available hopefully in the fall timeframe. But in the meantime, please enjoy James Urquhart talking 14 00:01:10,340 --> 00:01:16,260 about event-driven integration and event-driven architecture now. All right, James. So thanks for 15 00:01:16,260 --> 00:01:24,240 being on. Can you just start off by letting listeners in on what you mean when you're talking about event-driven 16 00:01:24,240 --> 00:01:28,140 integration, maybe even in comparison to like event-driven architecture or what the relationship 17 00:01:28,140 --> 00:01:37,600 is? Yeah, it's a great question. So event-driven architectures are really important and a growing 18 00:01:37,600 --> 00:01:47,280 phenomenon in large part because a lot of the problems that we have are about moving information through the 19 00:01:47,280 --> 00:01:53,180 processes and through the kind of the, you know, sort of the blood supply to the business and the blood supply to 20 00:01:53,180 --> 00:01:59,060 organizations in a way that's as effective and efficient as possible. So being able to move things so that you could 21 00:01:59,060 --> 00:02:05,120 react in real time when that data is available is a really powerful thing. And so event-driven architectures are built around 22 00:02:05,120 --> 00:02:13,020 sort of being able to react to data as it appears. And that appear obviously for it to appear to you, you have to know that 23 00:02:13,020 --> 00:02:18,880 that signal occurred, that involves transmitting things across the network, but to put that push model that event-driven 24 00:02:18,880 --> 00:02:24,280 architectures are driving are really powerful. And a great example I can give is there's a really brilliant 25 00:02:24,280 --> 00:02:31,480 presentation was given by a gentleman that used to work at Walmart at last year's DevOps Enterprise 26 00:02:31,480 --> 00:02:38,740 conference where he talked about Walmart's real-time inventory system and switching to an event-driven model 27 00:02:38,740 --> 00:02:43,660 for having, you know, instead of going and querying all the databases for all the different vendors that 28 00:02:43,660 --> 00:02:49,540 they were dealing with, they were asking to receive real-time updates of information. And then they would 29 00:02:49,540 --> 00:02:55,820 use that to update basically lookup tables for each application, which would make the, you know, all kinds of 30 00:02:55,820 --> 00:03:03,080 problems simpler around, you know, things around SLAs are easier and cheaper and being able to handle not having the 31 00:03:03,080 --> 00:03:12,100 correct up-to-date information is a little bit easier. But the key thing about that, you know, so that it really kind of 32 00:03:12,100 --> 00:03:19,600 drives a different economic model for a lot of problems. So with the Walmart example, one of the things that's really cool is that 33 00:03:19,600 --> 00:03:26,860 they're not just receiving Walmart data, right? Or they're receiving data from their suppliers as well. So they're 34 00:03:26,860 --> 00:03:34,240 receiving real-time updates of data from their suppliers. That's an integration problem, right? Because not all those guys are going to be using the same underlying 35 00:03:34,240 --> 00:03:43,600 platform that Walmart's using. And that opens up a real big question about what other data, business data, you know, device and environmental data, 36 00:03:43,600 --> 00:04:03,600 data, perhaps even economic data for sure, but political data potentially. What other forms of data would be better if we could get it to flow more efficiently through a market set of processes, right? The market system, not just an individual company's business problems. And so, you know, event-driven 37 00:04:03,600 --> 00:04:32,600 integration is really about using those event streams and using the technologies we have available today to stream information between entities to enable a more effective and efficient and easier to operate integration to enable that flow. It's about information that flows. It's not about things that you want to, you know, do a look up. I mean, that's a request response, right? It's about things where you want to say, I want to sit and listen and tell me when something's ready and I'll react to it. 38 00:04:32,600 --> 00:04:50,600 Okay. And, and what I'm hearing this, I'm thinking of, I mean, there were a lot of, and maybe this is very self-explanatory, but it might be worth elaborating on this. We definitely had data integration platforms previously, right? And then the whole goal there seemed to be like, it was similar. It seemed like a similar concept. 39 00:04:50,600 --> 00:05:19,600 Yeah. No, it's similar. I think the biggest difference between integrating via data at rest versus integrating via streams is there's a number of things you can do with streams a lot easier than you could do with data at rest. So data at rest is okay for updating, you know, a current state of something at any given time. 40 00:05:19,600 --> 00:05:37,060 Time as understood by that central data source. And everybody can read that and say, okay, I understand the current state, given the timeline I have between when I make the request and I get the response. So by the way, time is hugely important in all of this conversation, because that's where things go wrong. When you have race conditions and you have things like that. 41 00:05:37,600 --> 00:05:54,600 The problem with data at rest is getting, being able to replay any aspect of that, how that state came to be, or, or being able to have a vast number of things update that state at the same time is, is it becomes a scalability problem. That's really tough. 42 00:05:54,600 --> 00:05:55,600 That's really tough. 43 00:05:55,600 --> 00:06:13,600 Even for very modern databases to handle really well. They might be able to do a good job of sort of optimizing for say rights at the expense of some reads, where you might require a tremendous amount of infrastructure just for that central data store to be able to handle all the different types of loads that you want to put on things at scale. 44 00:06:13,600 --> 00:06:31,600 Distributed event processing when done very well is actually something where you can have a number of different things, be systems of records for their step in the process or their step in which they alter state. And you can replay events as they're, as they're captured from these systems. 45 00:06:31,600 --> 00:06:52,600 Basically you through a publish subscriber or a similar kind of method where you can say, look, me as a consumer of that data, I'm concerned about the stream as it comes from this entity. And I want, you know, to be able to understand the history of events and the current, the current set of events that I'm receiving from that entity and bring it across and then do something with it. 46 00:06:52,600 --> 00:07:07,600 I might store it into a data at rest so I can do analytics on it. I might try to do analytics in real time as it's coming in. I might trigger a, you know, another action or, or send a different event through a different stream to a different entity as a result of that coming in. 47 00:07:07,600 --> 00:07:16,600 But I, it's much more composable. And I wrote a blog post a long time ago about this. And in fact, I think you actually edited that blog post if I remember correctly. 48 00:07:16,600 --> 00:07:39,600 But yeah, but about the difference between composable and contextual systems and composable systems just in history have out survived and been, been much more flexible than contextual. Composable systems are ones like, you can think like the Linux command line, right? Where you can pipe information from one command to another, but you can assemble them any way you want. 49 00:07:39,600 --> 00:08:07,600 In contextual systems, you own the process, but you're assembling different workspaces that can work in there. Contextual systems are more like environments where you might have plugins, right? So that in that case, the environment owns the process and you're just plugging your work into the specific spots where you want to alter the work that's being done. Event-driven systems are much more composable than systems where you do call and response type of activities in a lot of cases. 50 00:08:07,600 --> 00:08:22,600 And certainly ones like, you know, some supply chain systems where they define the supply chain process and you're trying to plug your piece into that supply chain process. So that's the kind of the advantage there. And data-driven integration was a much more contextual thing. 51 00:08:22,600 --> 00:08:38,720 It was much more about the database setting the context for what the data is, how it can be queried, how it can be linked, how it can be consumed, and you having to, you know, essentially adapt what you're doing to the way the data was presented in that way. 52 00:08:38,720 --> 00:08:53,720 There are some overlaps there for sure. But I think event-driven is much more real-time and it's meant much more to be about only receiving the signal you want rather than you having to figure out how to pull the signal you want out of the data that's stored. 53 00:08:53,720 --> 00:09:06,260 All right. Fair. I mean, is this driven by like so many other things, just the proliferation of, I mean, it almost sounds cliche at this point, but smartphones and sensors and just all the connectivity and the nature and the scale of things. 54 00:09:06,260 --> 00:09:26,260 So because I mean, like we've talked about, I think we said, I mean, there's been data integration for a while, you know, API, you know, there's kind of this idea of API marketplaces and API integrations several years ago. Right. But it seems like, like the event-driven, even if event-driven architectures have been around for a long time, they seem to have picked up speed in the past couple or few years. 55 00:09:26,260 --> 00:09:38,660 And I'm wondering what the, like, what is it that's taking the discussion, you know, in that direction versus, you know, I don't hear as much about, again, like data integration or API integration or, or that sort of stuff as much now. 56 00:09:39,060 --> 00:09:55,840 Well, so first of all, let's be real clear. API integration absolutely has a place, you know, call response is really important. Data integration has a place where you potentially have multiple systems that want to use a specific data at rest system as, as a system of record. 57 00:09:56,260 --> 00:10:14,080 So it doesn't replace anything. The difference though, is that flow of information is that movement of information between systems in a way that it gets altered and then passed forward. And that you have multiple steps that data, an event may go through before it comes to rest. Right. And it may branch to multiple places and things like that. 58 00:10:14,140 --> 00:10:29,420 And the reason that's becoming a bigger deal now is in fact, the scale that you're talking about way more devices, way more, you know, both creators of events and way more, more sophisticated consumption of events across a numerous thing. 59 00:10:29,420 --> 00:10:49,780 So one way to look at it is a little bit like maybe, you know, a lot of complex systems adapt in very similar ways in which like a, like a plant might have a, you know, a trunk, a stem, you know, a stem and then a branches or limbs. And then, you know, then the individual leaves and a city might have major thoroughfares that run through it. 60 00:10:49,780 --> 00:10:56,840 And then there's, there's kind of the big commercial surface streets and then there's the neighborhood streets and even maybe alleyways and things like that. 61 00:10:57,320 --> 00:11:04,600 Those, but that evolutionary pattern is really common among systems that have something that flow through the system. 62 00:11:05,360 --> 00:11:19,120 And, and, and they, they, if there's a limited set of that flow and a limited set of, of things that can attach that flow, they tend to evolve in a way that they have that sort of trunk, limb, branch, leaf kind of scenario that comes out. 63 00:11:19,120 --> 00:11:30,740 And what you're seeing right now is attempting to make the flow more efficient and, and using an event-driven approach that helps a lot for the things that you have to react to in real time as soon as you possibly can. 64 00:11:31,040 --> 00:11:34,080 And you're seeing the evolution of the technologies around that. 65 00:11:34,140 --> 00:11:35,820 You hear edge a lot right now. 66 00:11:36,240 --> 00:11:42,580 Edge technologies are in big part taking advantage of event-driven in order to do the things that they do. 67 00:11:42,580 --> 00:11:48,500 And those edge systems are like limbs in a system where they have multiple branches and leaves hanging off of them. 68 00:11:48,760 --> 00:11:49,420 They're important. 69 00:11:49,540 --> 00:11:58,380 They're important because they isolate some of the processing in a way that you don't inundate central systems with every single piece of data and every single action necessary. 70 00:11:58,940 --> 00:12:09,060 And in fact, you know, I have a theory that's based a little bit off of some of the work that a gentleman named Jeffrey West did at the Santa Fe Institute around, you know, a big place for complex adaptive systems. 71 00:12:09,060 --> 00:12:15,920 And he says, you know, scale systems tend to evolve similar ways because of power laws and limitations in what can scale and what can't scale. 72 00:12:16,140 --> 00:12:25,720 And I have a theory that, you know, any given connection between two systems will have some sort of limit in terms of the amount of events and the amount of data you can pass via events between them. 73 00:12:25,800 --> 00:12:33,720 So that means the big central systems are going to have aggregated data in order to be able to handle the load that comes through. 74 00:12:33,780 --> 00:12:40,840 They can't get every event from every device handed all the way down through the trunk to everybody who wants to consume some aspect of that piece. 75 00:12:40,960 --> 00:12:42,500 You're going to have to have consolidation. 76 00:12:42,800 --> 00:12:50,100 You're going to have to have refactoring into a new form of event that means something in a slightly different context, but that has that contextual data where needed. 77 00:12:50,400 --> 00:13:00,220 You're going to have a bunch of problems along those lines where you're going to have this constant transformation of the information of the world as it moves through the different players and the different activities that have to happen. 78 00:13:00,220 --> 00:13:06,640 And I think absolutely it's being driven by the scale of I want it now. 79 00:13:06,780 --> 00:13:08,280 I want it to be accurate now. 80 00:13:08,440 --> 00:13:11,000 I want it to be able to react to what I want now. 81 00:13:11,100 --> 00:13:18,280 If I make a change in my checking account and there's a financial tracking system that I'm using that's attached to that account, 82 00:13:18,360 --> 00:13:27,080 I want that financial tracking system when I log into it a minute later to be absolutely up to date on what that transaction meant to my financial standing, right? 83 00:13:27,100 --> 00:13:29,540 Or whatever it is that it's analyzing. 84 00:13:30,080 --> 00:13:36,920 If I go in the healthcare system and I get a test done and the test is complete and that test gets information and I end up in an emergency room, 85 00:13:36,920 --> 00:13:49,140 I want the completion of that test to be readily available at the moment that it's completed for the doctor that's analyzing me rather than having to wait for a 24-hour batch processing cycle for that to be there. 86 00:13:49,140 --> 00:13:57,880 Yeah, the financial one is a great, I think, example that makes it really tangible for people is, right, if something happens and it's urgent to you, right? 87 00:13:57,920 --> 00:14:08,100 And the idea that you would have to wait for a system, right, especially if it's not like a daily batch to actually be alerted that something has changed and then maybe you can start having that discussion. 88 00:14:08,280 --> 00:14:09,320 Like, yeah, you want it now. 89 00:14:10,140 --> 00:14:17,760 So, yeah, I think that's, and by the way, I was going to say, Jeffrey West's book that James mentioned is aptly titled Scale. 90 00:14:19,360 --> 00:14:25,800 And it is really interesting and informationally dense, but it's probably worth checking out. 91 00:14:26,140 --> 00:14:26,680 Yeah, it is. 92 00:14:26,860 --> 00:14:39,000 And that concept of power laws, if you do get into the scientist-y, you know, tech-y thing about it a little bit more, that concept of power laws is really, really a powerful concept to understand in terms of how software systems scale. 93 00:14:39,000 --> 00:14:50,900 And, you know, because our processing per transistor on the chip isn't changing anytime soon, or per unit of, you know, per CPU core on a chip isn't changing anytime soon. 94 00:14:51,200 --> 00:14:53,440 Speed of light certainly isn't changing anytime soon. 95 00:14:53,600 --> 00:14:57,540 So we're scaling like crazy around things that don't scale any further. 96 00:14:57,760 --> 00:15:07,980 And so that means we have other constructs that have to come into play to allow us to use those resources at their full potential while still being able to scale well beyond what's capable. 97 00:15:08,720 --> 00:15:08,840 Yeah. 98 00:15:09,500 --> 00:15:11,420 So what do you think is the biggest change that has to happen? 99 00:15:11,500 --> 00:15:18,960 I mean, if I'm a CIO or if I'm whomever within an organization saying, okay, you know, we've got to move into a venture of an architecture, then a venture of an integration. 100 00:15:19,160 --> 00:15:29,080 Like where, I mean, is it the data systems, the app architectures, like where are the biggest pain points or the biggest points that should be addressed if you're trying to move in this direction? 101 00:15:29,640 --> 00:15:34,260 Yeah, I think, you know, I always say that my biggest thing is understand what has to flow, right? 102 00:15:34,260 --> 00:15:41,060 Begin to think of your systems in terms of the flow of information rather than in terms of the user interface talking to a database. 103 00:15:42,000 --> 00:15:49,560 You know, we've had this sort of big three-tier model kind of, okay, every problem is there's a database and there's a UI and the two talk to each other. 104 00:15:49,960 --> 00:15:55,380 Begin to think about, and a lot of, you know, enterprise architects already do this, 105 00:15:55,460 --> 00:16:09,140 but begin to think about the flow of information that you need from any event that occurs that you want to be able to respond to and get value from to any other, you know, any other point in your data, in your world where that has to link to. 106 00:16:09,140 --> 00:16:22,820 I would say that, you know, if you, rather than having the traditional object graph that you had before about your environment, if you had a data flow graph, a directed graph understating of your systems architecture in your company, 107 00:16:22,820 --> 00:16:27,620 it would be a really valuable and eye-opening thing to be able to take a look at. 108 00:16:27,720 --> 00:16:35,860 And then from there, you can begin to look at which of these data sets need to be a request response, which of these interactions or request responses should be API driven, 109 00:16:35,860 --> 00:16:41,540 which of these interactions should be about receiving information for as a stream from something else. 110 00:16:41,680 --> 00:16:50,120 And, you know, are there pieces of information where, you know what, like a lot of analytics systems, just, you know, have a data warehouse or a data lake and point your analytics systems at that. 111 00:16:50,180 --> 00:16:52,300 That's actually not a bad model in those situations. 112 00:16:52,300 --> 00:17:01,340 You don't have to process the stream directly to get your analytics unless you need real-time information about a specific aspect of what you're trying to deal with. 113 00:17:02,000 --> 00:17:10,440 Observability is a good one for that, you know, good understanding that, or if you're running campaigns in a retail environment and you want to understand how's that campaign going? 114 00:17:10,520 --> 00:17:12,920 How are people responding to the campaign in real time? 115 00:17:13,200 --> 00:17:17,320 Those are the kinds of situations where you might want to process the stream right away. 116 00:17:17,460 --> 00:17:23,780 But in other situations, you're looking at what's the last month's, you know, report for X. 117 00:17:24,020 --> 00:17:26,540 In those situations, you may not need to do that. 118 00:17:26,600 --> 00:17:33,300 But as you get that understanding of how information flows and looking at it in terms of the flow of information to the business, 119 00:17:33,420 --> 00:17:38,540 then I think the next thing that you do is you begin to look at your application portfolio and understanding 120 00:17:38,540 --> 00:17:45,280 where would it make the most sense to refactor what we're doing from, you know, a call tree, 121 00:17:45,280 --> 00:17:53,140 like a services call tree to something that's more an event-driven architecture where maybe you're receiving the data 122 00:17:53,140 --> 00:17:57,600 and building your own kind of local representation of the state from that. 123 00:17:57,700 --> 00:18:01,500 And there's also systems that do, by the way, huge amounts of state at scale. 124 00:18:01,680 --> 00:18:09,020 Swim.ai is an interesting platform for that purpose where you could actually build a representational state model 125 00:18:09,020 --> 00:18:16,000 of a very complex system right there in memory on a surprisingly few number of systems from a scale perspective. 126 00:18:16,600 --> 00:18:22,600 And then take, you know, basically build either AI models or other kind of rule-based models 127 00:18:22,600 --> 00:18:26,900 that can respond to those state situations on an agent-by-agent basis. 128 00:18:26,900 --> 00:18:32,400 So each individual element can figure out how it responds to the world around it and take action. 129 00:18:33,000 --> 00:18:37,300 And so you can do, there's some really interesting new models that are out there that are worth looking at 130 00:18:37,300 --> 00:18:38,980 and seeing if they make your situation better. 131 00:18:39,060 --> 00:18:42,480 But in terms of your legacy infrastructure, you can also look at, you know, 132 00:18:42,540 --> 00:18:47,600 are there ways that I can take a data update and turn it into an event, right? 133 00:18:47,840 --> 00:18:54,160 Are there ways that I can, instead of doing a query on somebody else's database 134 00:18:54,160 --> 00:18:57,880 or calling a microservice even with a request response, 135 00:18:57,880 --> 00:19:01,760 is there a way that I could turn that around and have a microservice send me an event 136 00:19:01,760 --> 00:19:05,460 that I can just sit and listen to and respond to when it arrives? 137 00:19:05,860 --> 00:19:12,280 Those are the questions to begin to understand and begin to understand how to modify the flow of information in your world 138 00:19:12,280 --> 00:19:16,100 so that it's as directed in the right direction as possible. 139 00:19:16,100 --> 00:19:21,120 And there's a minimum number of conversational elements that have to happen in order to enable it. 140 00:19:21,920 --> 00:19:25,560 So when I'm hearing this, two specific technologies come to mind. 141 00:19:25,640 --> 00:19:28,380 One is like serverless slash functions. 142 00:19:28,820 --> 00:19:32,320 And the other is Kafka or something, you know, along those lines. 143 00:19:32,420 --> 00:19:35,600 I think they call Kafka a message broker is probably unfair at this point. 144 00:19:35,860 --> 00:19:41,660 But, you know, and RabbitMQ or something, you know, along those lines is probably in there as well. 145 00:19:41,960 --> 00:19:46,540 So what are the roles of, you know, some of these technologies in this? 146 00:19:47,360 --> 00:19:53,000 I mean, is this where, you know, serverless or Kafka or anything like this becomes kind of a very commonplace? 147 00:19:53,440 --> 00:19:56,280 Yeah, well, I'm actually kind of writing a book on this topic right now. 148 00:19:56,420 --> 00:20:01,540 So if people can kind of look for that this fall, hopefully by the end of the year, it should be out. 149 00:20:01,540 --> 00:20:08,160 But there's a set of components that come together to kind of make the full end and piece of flow, right? 150 00:20:08,220 --> 00:20:15,180 The most important thing to kind of look at is right now, just making an integration work involves figuring out 151 00:20:15,180 --> 00:20:18,880 what somebody else's interface is for establishing the connection. 152 00:20:19,380 --> 00:20:23,080 What's the metadata they send on the stream that describes what the event is 153 00:20:23,080 --> 00:20:24,000 and the context of the event? 154 00:20:24,100 --> 00:20:26,820 What's the payload of the event and how to interpret the payload event? 155 00:20:27,240 --> 00:20:31,040 I think the first thing to understand is, you know, plan for a future 156 00:20:31,040 --> 00:20:34,900 in which that stuff becomes more and more commodity and standardized. 157 00:20:35,160 --> 00:20:38,620 Plan for a future in which what you want to do is make sure you understand your stream 158 00:20:38,620 --> 00:20:43,160 and you're looking for standards to consume in order to package events on that stream. 159 00:20:43,300 --> 00:20:46,900 There's a cloud event standard that CNCF has put out. 160 00:20:47,420 --> 00:20:48,420 That's a good thing to look at. 161 00:20:48,460 --> 00:20:49,500 Maybe that's the right option. 162 00:20:50,060 --> 00:20:55,580 MQTP, MQTT, these are all, you know, things, technologies that people who are kind of dealing 163 00:20:55,580 --> 00:20:58,640 with integration are probably somewhat aware of already. 164 00:20:59,200 --> 00:21:03,960 So you need that connection and you need that ability, those interfaces and protocols to make 165 00:21:03,960 --> 00:21:05,000 that connection work. 166 00:21:05,220 --> 00:21:08,220 And then you need, at the other end, you need processing capabilities. 167 00:21:08,220 --> 00:21:13,640 And that's where something like Kafka comes into play or swim.ai that I mentioned earlier 168 00:21:13,640 --> 00:21:16,740 as well as a different version, a different kind of view of that. 169 00:21:16,860 --> 00:21:23,160 So Kafka has evolved from being strictly a queue, a log-based queue, and a collector of events 170 00:21:23,160 --> 00:21:28,460 and something that can handle high, high volumes of events coming in and putting them in a mechanism 171 00:21:28,460 --> 00:21:33,640 that you could use to read and understand those events to having some processing capability 172 00:21:33,640 --> 00:21:35,300 in addition to that, right? 173 00:21:35,340 --> 00:21:37,240 So it becomes more of a platform kind of view. 174 00:21:37,780 --> 00:21:45,140 But, and Rabbit as well is introducing log-based queuing as, in addition to message-based queuing. 175 00:21:45,240 --> 00:21:49,940 So queues are very important because you want to decouple the synchronicity. 176 00:21:50,500 --> 00:21:51,860 We talked about how important time is. 177 00:21:51,900 --> 00:21:56,360 You want to decouple that synchronicity between producer and consumer as much as possible. 178 00:21:56,960 --> 00:21:58,180 But there's other things too, right? 179 00:21:58,260 --> 00:22:00,640 There's the, you know, what are the sources? 180 00:22:00,940 --> 00:22:01,740 What are the devices? 181 00:22:02,080 --> 00:22:04,840 What are the, you know, where are you getting your events from? 182 00:22:04,840 --> 00:22:08,580 What are the technologies you're going to use to generate those events and package them correctly 183 00:22:08,580 --> 00:22:11,500 and put them out there and into the connection? 184 00:22:11,880 --> 00:22:12,820 And what are the syncs? 185 00:22:13,180 --> 00:22:17,680 What are the things that are going to be sort of the final resting places or final actions 186 00:22:17,680 --> 00:22:20,940 that an event goes through on any given branch that it takes? 187 00:22:21,020 --> 00:22:25,340 Because events at some point in time, the flow stops, right? 188 00:22:25,420 --> 00:22:28,120 And it becomes data at rest that can be analyzed later. 189 00:22:28,720 --> 00:22:33,460 And so there's all kinds of things there around, you know, if you look at just AWS's ecosystem 190 00:22:33,460 --> 00:22:37,160 alone, or if you look at even, you know, to a certain extent, some of the stuff that we've 191 00:22:37,160 --> 00:22:42,340 done with TAS and Tansu Application Service and others, there's a number of things you can 192 00:22:42,340 --> 00:22:48,460 kind of do with, you know, already today with having events come into being displayed in a 193 00:22:48,460 --> 00:22:55,380 visualization dashboard, for instance, or being used to trigger actions, both in a digital 194 00:22:55,380 --> 00:22:56,920 way or in a physical way. 195 00:22:56,980 --> 00:23:00,780 And, you know, you can have machines respond or robots respond to events as they come in. 196 00:23:01,100 --> 00:23:02,300 So those are all very possible. 197 00:23:02,360 --> 00:23:03,500 And you mentioned serverless. 198 00:23:03,600 --> 00:23:08,480 I think, you know, having really, really efficient stream-based processing is super important. 199 00:23:08,480 --> 00:23:11,820 And I, you know, I love Lambda as one aspect of that. 200 00:23:11,820 --> 00:23:15,500 I love something like step functions as one aspect of that. 201 00:23:15,600 --> 00:23:19,200 And each of the cloud providers has their sort of versions of various aspects of that. 202 00:23:19,540 --> 00:23:26,060 But you have, you know, the ability to quickly take an action and make a decision or signal 203 00:23:26,060 --> 00:23:28,240 something else with a new event or whatever. 204 00:23:28,360 --> 00:23:32,040 Those things are going to be the nodes in the graph. 205 00:23:32,460 --> 00:23:34,820 So that's the really important part of it. 206 00:23:35,560 --> 00:23:40,280 Anyway, so that's what I say is the critical, are the critical elements to think of there. 207 00:23:40,340 --> 00:23:43,400 But there's a lot of, you know, I talk about a lot of different components that are really 208 00:23:43,400 --> 00:23:43,740 important. 209 00:23:43,740 --> 00:23:48,740 And the underlying, the cloud makes it possible in a lot of ways, because having a really 210 00:23:48,740 --> 00:23:52,620 reliable infrastructure that you know is going to run a certain way that's easily consumable 211 00:23:52,620 --> 00:23:56,300 at different scales is one of the enablers that makes this happen. 212 00:23:56,580 --> 00:24:00,260 And, and that's, that's both sort of, you know, public cloud, private cloud, you know, 213 00:24:00,260 --> 00:24:06,000 all those aspects, but be having that, not having firmly and fixed and coupled assigned 214 00:24:06,000 --> 00:24:11,620 hardware and, and network routes and things like that, but being able to reuse things much 215 00:24:11,620 --> 00:24:13,460 more easily is a big part of this as well. 216 00:24:14,060 --> 00:24:14,200 Yeah. 217 00:24:14,260 --> 00:24:18,140 And I mean, I just at a much higher level, I, I often go back to this idea of like, why, 218 00:24:18,420 --> 00:24:20,000 like, why are we talking about a lot of things now? 219 00:24:20,000 --> 00:24:23,320 And part of it is like the part of things, like we've been talking about certain things 220 00:24:23,320 --> 00:24:27,380 for years, but now we have the pieces in place to actually do it. 221 00:24:27,700 --> 00:24:27,840 Yeah. 222 00:24:27,940 --> 00:24:31,300 And it often starts with the cloud and starts with some of these data systems and the, all 223 00:24:31,300 --> 00:24:32,400 these, all these pieces. 224 00:24:32,400 --> 00:24:37,640 And then you can actually start to realize some of the, you know, maybe utopia and they're 225 00:24:37,640 --> 00:24:40,980 kind of, you know, big picture things that we've actually been talking about for a long 226 00:24:40,980 --> 00:24:41,200 time. 227 00:24:41,300 --> 00:24:42,480 So I think you're right. 228 00:24:42,540 --> 00:24:44,300 Like the cloud is an integral part. 229 00:24:44,400 --> 00:24:49,980 I'm just one more question, James, which is like, it seems like most or many people, 230 00:24:49,980 --> 00:24:53,760 many discussions come down to a matter of, you know, as we're talking about the technological 231 00:24:53,760 --> 00:24:58,420 stuff and the architectural aspects, it also comes into like the organizational part of 232 00:24:58,420 --> 00:24:58,600 things. 233 00:24:58,640 --> 00:25:02,700 And I remember we did a collection of blog posts with discover last year. 234 00:25:02,700 --> 00:25:06,840 And one part of it was the, the enterprise driven architecture that, you know, on the product 235 00:25:06,840 --> 00:25:10,420 side, but like behind the scenes, there was also some organizational changes and process 236 00:25:10,420 --> 00:25:16,180 changes and, and all these other quote unquote digital transformation activities going on. 237 00:25:16,180 --> 00:25:21,180 So how do you, when you look at like adventure of an integration, just the adventure of an 238 00:25:21,180 --> 00:25:26,660 architecture at large, how do you think that plays into, you know, some of the discussions 239 00:25:26,660 --> 00:25:30,720 about organizational changes and processes and teams and, and what's the connection? 240 00:25:31,720 --> 00:25:31,840 Yeah. 241 00:25:32,100 --> 00:25:37,920 One of the things I think is really interesting about the last 10 years is the, the organizational 242 00:25:37,920 --> 00:25:46,120 models that we used when we were being very contextual in the way we built IT, meaning, you know, 243 00:25:46,140 --> 00:25:51,360 we had fixed resources, fixed ability to get things done. 244 00:25:51,620 --> 00:25:56,840 We had, you know, we had to be very, very careful in how we plan to use resources because they were 245 00:25:56,840 --> 00:26:00,840 incredibly expensive and incredibly difficult to even acquire and get installed. 246 00:26:01,600 --> 00:26:07,840 So we were very kind of construction oriented in the way we thought about software for a long time. 247 00:26:07,940 --> 00:26:13,860 And the last 15 years or so have really been about this shift to breaking out of that construction 248 00:26:13,860 --> 00:26:17,940 mold and getting more into what I call, you know, a more curated mode, right? 249 00:26:17,940 --> 00:26:25,140 More of like a gardener or more of a zookeeper in the sense that, you know, yeah, we have some 250 00:26:25,140 --> 00:26:26,560 fixed elements about what we're doing. 251 00:26:26,640 --> 00:26:31,100 We have some fixed constraints, but what we really want to do is help a system evolve that 252 00:26:31,100 --> 00:26:35,660 gives the kind of positive feedback loops that we want and eliminates as many of the negative 253 00:26:35,660 --> 00:26:37,320 feedback loops that we don't want. 254 00:26:37,740 --> 00:26:41,160 We want to shape and evolve and adapt as we go. 255 00:26:41,300 --> 00:26:44,940 And so the problems around event-driven architectures are very much like the problems around most 256 00:26:44,940 --> 00:26:49,160 modern application and, you know, cloud-native modern application development, whatever you 257 00:26:49,160 --> 00:26:54,200 want to call it, in that now really what you want is a much more flexible organization in 258 00:26:54,200 --> 00:26:59,420 terms of how you develop, but you want it to be built in an environment that can enable 259 00:26:59,420 --> 00:27:03,420 that flexibility and provide a contextual kind of platform. 260 00:27:03,960 --> 00:27:07,440 And so, you know, I always say it's really important if you have infrastructure you have to 261 00:27:07,440 --> 00:27:09,680 run, then absolutely you need an infrastructure team. 262 00:27:09,840 --> 00:27:14,620 Understand that their focus really is about delivering the capacity that you need to run 263 00:27:14,620 --> 00:27:15,060 software. 264 00:27:15,460 --> 00:27:20,460 Their job is to deliver the storage network and compute that you need in order to make a 265 00:27:20,460 --> 00:27:21,540 software system run. 266 00:27:21,860 --> 00:27:23,700 So that's, that's, but that's super important. 267 00:27:23,960 --> 00:27:28,920 And so if you have data centers and most enterprises do that, it's critical to have a really good 268 00:27:28,920 --> 00:27:33,620 infrastructure team, making sure that they understand where the borderline is of what 269 00:27:33,620 --> 00:27:34,180 they're delivering. 270 00:27:35,100 --> 00:27:39,160 So the next element then is, you know, you have development teams and those are highly 271 00:27:39,160 --> 00:27:43,880 agile and you really have this problem today of how do you not organize in a way that your, 272 00:27:43,880 --> 00:27:50,400 your, your development teams are locked into certain organizations and certain activities. 273 00:27:50,580 --> 00:27:56,040 You want some ability, at least within a business unit, to be able to quickly adapt how resources 274 00:27:56,040 --> 00:27:59,400 are allocated to meet whatever the priorities are at the time. 275 00:27:59,400 --> 00:28:04,320 And so I think there's a lot of practices around pair programming and other things. 276 00:28:04,520 --> 00:28:07,900 It doesn't have to be pair programming, but pair programming helps with you being able 277 00:28:07,900 --> 00:28:11,020 to reassign people across teams a little bit more easily. 278 00:28:11,500 --> 00:28:15,000 You don't necessarily have to change their management, but you absolutely can change their 279 00:28:15,000 --> 00:28:18,420 assignments on a more regular basis and make it a little bit easier for them. 280 00:28:18,460 --> 00:28:21,960 And in the middle there, then, you know, what developers care about is getting their stuff 281 00:28:21,960 --> 00:28:22,580 into production. 282 00:28:22,740 --> 00:28:24,520 They don't care about finding capacity. 283 00:28:24,680 --> 00:28:26,300 That's the last thing they want to worry about. 284 00:28:26,300 --> 00:28:29,720 They just want to get their stuff in front of customers and start getting feedback from 285 00:28:29,720 --> 00:28:30,160 customers. 286 00:28:31,080 --> 00:28:34,320 And so from that perspective, then the last thing you need is a good path to production 287 00:28:34,320 --> 00:28:35,000 for developers. 288 00:28:35,180 --> 00:28:39,260 And to me, that's where the platform team fits in the middle, like firmly between helping 289 00:28:39,260 --> 00:28:43,540 understand how infrastructure gets consumed and working with the infrastructure team on making 290 00:28:43,540 --> 00:28:49,520 sure that software gets assigned to capacity in a, in a smart and intelligent way, but also 291 00:28:49,520 --> 00:28:54,160 at the same time, listening to developers and working with developers to eliminate as many bottlenecks 292 00:28:54,160 --> 00:28:59,640 in that process and many constraints in the development process as you can and working systematically at 293 00:28:59,640 --> 00:29:03,260 improving that development process over time as much as possible. 294 00:29:03,260 --> 00:29:08,840 And so I look at sort of that platform team element as being a really important organizational 295 00:29:08,840 --> 00:29:15,080 piece for providing standard building blocks like databases and messaging systems and, you 296 00:29:15,080 --> 00:29:20,240 know, all the Kafka stuff we talked about, all the event-driven pieces, all the underlying services 297 00:29:20,240 --> 00:29:24,660 that enable an application team to build something that's business specific and flow information 298 00:29:24,660 --> 00:29:27,100 between themselves and other systems out there. 299 00:29:27,340 --> 00:29:31,760 And, you know, and also being able to provide the building blocks for actually writing code, 300 00:29:32,020 --> 00:29:38,340 providing the, you know, kind of trusted software supply chain that makes code easy to execute 301 00:29:38,340 --> 00:29:43,240 and makes code development easy to execute and getting to production something that security 302 00:29:43,240 --> 00:29:43,620 can go. 303 00:29:43,700 --> 00:29:46,880 Yeah, we're really comfortable if they build this certain way that what will come out the 304 00:29:46,880 --> 00:29:48,080 other end will pass on it. 305 00:29:48,800 --> 00:29:52,980 And so those things are really important and pieces that I think an organization, that's 306 00:29:52,980 --> 00:29:56,600 why an organization really needs to look at platforms and they really need to look at platform 307 00:29:56,600 --> 00:30:01,680 teams because even, you know, I, I ideally want them to use Tanzu application service because 308 00:30:01,680 --> 00:30:05,900 we think just makes it a billion times easier than doing just about anything else out there. 309 00:30:06,260 --> 00:30:09,160 But let's say that they don't want a platform like that. 310 00:30:09,160 --> 00:30:13,280 Well, at the very least, you need an organization that's worried about building the right path 311 00:30:13,280 --> 00:30:18,340 to production for your developers and simplifying, getting code to production as much as possible. 312 00:30:18,340 --> 00:30:23,460 And they could do it all with open source or all with something else, but it's still a need. 313 00:30:23,660 --> 00:30:27,980 And it's still something that I think, you know, we advise our executives on our exec customer 314 00:30:27,980 --> 00:30:33,040 executives on a regular basis that they really need to look at the role of the platform team 315 00:30:33,040 --> 00:30:36,480 as being that bridge from capacity to a path to production. 316 00:30:37,220 --> 00:30:37,800 All right. 317 00:30:38,880 --> 00:30:39,940 Insightful as always, James. 318 00:30:40,000 --> 00:30:40,460 Thanks a lot. 319 00:30:41,040 --> 00:30:41,900 I appreciate it, Derek. 320 00:30:41,980 --> 00:30:42,740 Good to talk to you, man. 321 00:30:45,040 --> 00:30:45,680 All right. 322 00:30:45,800 --> 00:30:46,520 There you have it. 323 00:30:46,580 --> 00:30:47,740 I hope you learned something. 324 00:30:48,340 --> 00:30:49,260 I certainly did. 325 00:30:49,560 --> 00:30:55,460 To follow James Urquhart, you can check him out, follow him on Twitter at James Urquhart. 326 00:30:56,460 --> 00:30:59,000 That's James spelled how you would normally spell James. 327 00:30:59,140 --> 00:31:02,620 And then U-R-Q-U-H-A-R-T. 328 00:31:03,300 --> 00:31:08,680 And to follow, to hear more about VMware's myriad toolings around cloud-native architectures 329 00:31:08,680 --> 00:31:15,480 and applications and platforms and event-driven architecture, just check out tanzu.vmware.com. 330 00:31:15,960 --> 00:31:20,220 Tanzu, T-A-N-Z-U.vmware.com. 331 00:31:20,520 --> 00:31:21,480 Thanks for listening.