Event-driven integration and architecture (with James Urquhart)

Episode 218 · Tue, 23 Jun 2020 14:00:00 +0200 · 31:25 · SRT · archive.org · all Tanzu Talk

Welcome to the Cloud Native in 15 Minutes podcast. I'm your host, Derek Harris from VMware. And I'll skip the long intro I used to give because, well, this podcast is not 15 minutes, it's closer to 30 minutes. But if you listen to it at 2x speed, you'll about hit that 15-minute mark. So joining me, my guest today is James Urquhart of VMware's Strategic Executive Advisory Office. And James, I guess he's a Strategic Executive Advisor. And James talks a lot, he's very knowledgeable about distributed systems. And in this particular podcast, we talk a lot about event-driven integration and event-driven architecture and why that is where a lot of applications are heading and why it's valuable in a world where you want real-time insights and being able to act in real time on events that are happening and the tools and the processes and the organizational changes that need to happen to make that happen. So I think it's a really, really insightful interview. James is a really insightful person on this topic and other topics. And he's writing a book about event-driven integration that he references. So that should be available hopefully in the fall timeframe. But in the meantime, please enjoy James Urquhart talking about event-driven integration and event-driven architecture now. All right, James. So thanks for being on. Can you just start off by letting listeners in on what you mean when you're talking about event-driven integration, maybe even in comparison to like event-driven architecture or what the relationship is? Yeah, it's a great question. So event-driven architectures are really important and a growing phenomenon in large part because a lot of the problems that we have are about moving information through the processes and through the kind of the, you know, sort of the blood supply to the business and the blood supply to organizations in a way that's as effective and efficient as possible. So being able to move things so that you could react in real time when that data is available is a really powerful thing. And so event-driven architectures are built around

sort of being able to react to data as it appears. And that appear obviously for it to appear to you, you have to know that that signal occurred, that involves transmitting things across the network, but to put that push model that event-driven architectures are driving are really powerful. And a great example I can give is there's a really brilliant presentation was given by a gentleman that used to work at Walmart at last year's DevOps Enterprise conference where he talked about Walmart's real-time inventory system and switching to an event-driven model for having, you know, instead of going and querying all the databases for all the different vendors that they were dealing with, they were asking to receive real-time updates of information. And then they would use that to update basically lookup tables for each application, which would make the, you know, all kinds of problems simpler around, you know, things around SLAs are easier and cheaper and being able to handle not having the correct up-to-date information is a little bit easier. But the key thing about that, you know, so that it really kind of drives a different economic model for a lot of problems. So with the Walmart example, one of the things that's really cool is that they're not just receiving Walmart data, right? Or they're receiving data from their suppliers as well. So they're receiving real-time updates of data from their suppliers. That's an integration problem, right? Because not all those guys are going to be using the same underlying platform that Walmart's using. And that opens up a real big question about what other data, business data, you know, device and environmental data, data, perhaps even economic data for sure, but political data potentially. What other forms of data would be better if we could get it to flow more efficiently through a market set of processes, right? The market system, not just an individual company's business problems. And so, you know, event-driven integration is really about using those event streams and using the technologies we have available today to stream information between entities to enable a more effective and efficient and easier to operate integration to enable that flow. It's about information that flows. It's not about things that you want to, you know, do a look up. I mean, that's a request response, right? It's about things where you want to say, I want to sit and listen and tell me when something's ready and I'll react to it.

Okay. And, and what I'm hearing this, I'm thinking of, I mean, there were a lot of, and maybe this is very self-explanatory, but it might be worth elaborating on this. We definitely had data integration platforms previously, right? And then the whole goal there seemed to be like, it was similar. It seemed like a similar concept. Yeah. No, it's similar. I think the biggest difference between integrating via data at rest versus integrating via streams is there's a number of things you can do with streams a lot easier than you could do with data at rest. So data at rest is okay for updating, you know, a current state of something at any given time. Time as understood by that central data source. And everybody can read that and say, okay, I understand the current state, given the timeline I have between when I make the request and I get the response. So by the way, time is hugely important in all of this conversation, because that's where things go wrong. When you have race conditions and you have things like that. The problem with data at rest is getting, being able to replay any aspect of that, how that state came to be, or, or being able to have a vast number of things update that state at the same time is, is it becomes a scalability problem. That's really tough. That's really tough. Even for very modern databases to handle really well. They might be able to do a good job of sort of optimizing for say rights at the expense of some reads, where you might require a tremendous amount of infrastructure just for that central data store to be able to handle all the different types of loads that you want to put on things at scale. Distributed event processing when done very well is actually something where you can have a number of different things, be systems of records for their step in the process or their step in which they alter state. And you can replay events as they're, as they're captured from these systems. Basically you through a publish subscriber or a similar kind of method where you can say, look, me as a consumer of that data, I'm concerned about the stream as it comes from this entity. And I want, you know, to be able to understand the history of events and the current, the current set of events that I'm receiving from that entity and bring it across and then do something with it.

I might store it into a data at rest so I can do analytics on it. I might try to do analytics in real time as it's coming in. I might trigger a, you know, another action or, or send a different event through a different stream to a different entity as a result of that coming in. But I, it's much more composable. And I wrote a blog post a long time ago about this. And in fact, I think you actually edited that blog post if I remember correctly. But yeah, but about the difference between composable and contextual systems and composable systems just in history have out survived and been, been much more flexible than contextual. Composable systems are ones like, you can think like the Linux command line, right? Where you can pipe information from one command to another, but you can assemble them any way you want. In contextual systems, you own the process, but you're assembling different workspaces that can work in there. Contextual systems are more like environments where you might have plugins, right? So that in that case, the environment owns the process and you're just plugging your work into the specific spots where you want to alter the work that's being done. Event-driven systems are much more composable than systems where you do call and response type of activities in a lot of cases. And certainly ones like, you know, some supply chain systems where they define the supply chain process and you're trying to plug your piece into that supply chain process. So that's the kind of the advantage there. And data-driven integration was a much more contextual thing. It was much more about the database setting the context for what the data is, how it can be queried, how it can be linked, how it can be consumed, and you having to, you know, essentially adapt what you're doing to the way the data was presented in that way. There are some overlaps there for sure. But I think event-driven is much more real-time and it's meant much more to be about only receiving the signal you want rather than you having to figure out how to pull the signal you want out of the data that's stored.

All right. Fair. I mean, is this driven by like so many other things, just the proliferation of, I mean, it almost sounds cliche at this point, but smartphones and sensors and just all the connectivity and the nature and the scale of things. So because I mean, like we've talked about, I think we said, I mean, there's been data integration for a while, you know, API, you know, there's kind of this idea of API marketplaces and API integrations several years ago. Right. But it seems like, like the event-driven, even if event-driven architectures have been around for a long time, they seem to have picked up speed in the past couple or few years. And I'm wondering what the, like, what is it that's taking the discussion, you know, in that direction versus, you know, I don't hear as much about, again, like data integration or API integration or, or that sort of stuff as much now. Well, so first of all, let's be real clear. API integration absolutely has a place, you know, call response is really important. Data integration has a place where you potentially have multiple systems that want to use a specific data at rest system as, as a system of record. So it doesn't replace anything. The difference though, is that flow of information is that movement of information between systems in a way that it gets altered and then passed forward. And that you have multiple steps that data, an event may go through before it comes to rest. Right. And it may branch to multiple places and things like that. And the reason that's becoming a bigger deal now is in fact, the scale that you're talking about way more devices, way more, you know, both creators of events and way more, more sophisticated consumption of events across a numerous thing. So one way to look at it is a little bit like maybe, you know, a lot of complex systems adapt in very similar ways in which like a, like a plant might have a, you know, a trunk, a stem, you know, a stem and then a branches or limbs. And then, you know, then the individual leaves and a city might have major thoroughfares that run through it. And then there's, there's kind of the big commercial surface streets and then there's the neighborhood streets and even maybe alleyways and things like that.

Those, but that evolutionary pattern is really common among systems that have something that flow through the system. And, and, and they, they, if there's a limited set of that flow and a limited set of, of things that can attach that flow, they tend to evolve in a way that they have that sort of trunk, limb, branch, leaf kind of scenario that comes out. And what you're seeing right now is attempting to make the flow more efficient and, and using an event-driven approach that helps a lot for the things that you have to react to in real time as soon as you possibly can. And you're seeing the evolution of the technologies around that. You hear edge a lot right now. Edge technologies are in big part taking advantage of event-driven in order to do the things that they do. And those edge systems are like limbs in a system where they have multiple branches and leaves hanging off of them. They're important. They're important because they isolate some of the processing in a way that you don't inundate central systems with every single piece of data and every single action necessary. And in fact, you know, I have a theory that's based a little bit off of some of the work that a gentleman named Jeffrey West did at the Santa Fe Institute around, you know, a big place for complex adaptive systems. And he says, you know, scale systems tend to evolve similar ways because of power laws and limitations in what can scale and what can't scale. And I have a theory that, you know, any given connection between two systems will have some sort of limit in terms of the amount of events and the amount of data you can pass via events between them. So that means the big central systems are going to have aggregated data in order to be able to handle the load that comes through. They can't get every event from every device handed all the way down through the trunk to everybody who wants to consume some aspect of that piece. You're going to have to have consolidation. You're going to have to have refactoring into a new form of event that means something in a slightly different context, but that has that contextual data where needed. You're going to have a bunch of problems along those lines where you're going to have this constant transformation of the information of the world as it moves through the different players and the different activities that have to happen.

And I think absolutely it's being driven by the scale of I want it now. I want it to be accurate now. I want it to be able to react to what I want now. If I make a change in my checking account and there's a financial tracking system that I'm using that's attached to that account, I want that financial tracking system when I log into it a minute later to be absolutely up to date on what that transaction meant to my financial standing, right? Or whatever it is that it's analyzing. If I go in the healthcare system and I get a test done and the test is complete and that test gets information and I end up in an emergency room, I want the completion of that test to be readily available at the moment that it's completed for the doctor that's analyzing me rather than having to wait for a 24-hour batch processing cycle for that to be there. Yeah, the financial one is a great, I think, example that makes it really tangible for people is, right, if something happens and it's urgent to you, right? And the idea that you would have to wait for a system, right, especially if it's not like a daily batch to actually be alerted that something has changed and then maybe you can start having that discussion. Like, yeah, you want it now. So, yeah, I think that's, and by the way, I was going to say, Jeffrey West's book that James mentioned is aptly titled Scale. And it is really interesting and informationally dense, but it's probably worth checking out. Yeah, it is. And that concept of power laws, if you do get into the scientist-y, you know, tech-y thing about it a little bit more, that concept of power laws is really, really a powerful concept to understand in terms of how software systems scale. And, you know, because our processing per transistor on the chip isn't changing anytime soon, or per unit of, you know, per CPU core on a chip isn't changing anytime soon. Speed of light certainly isn't changing anytime soon. So we're scaling like crazy around things that don't scale any further. And so that means we have other constructs that have to come into play to allow us to use those resources at their full potential while still being able to scale well beyond what's capable.

Yeah. So what do you think is the biggest change that has to happen? I mean, if I'm a CIO or if I'm whomever within an organization saying, okay, you know, we've got to move into a venture of an architecture, then a venture of an integration. Like where, I mean, is it the data systems, the app architectures, like where are the biggest pain points or the biggest points that should be addressed if you're trying to move in this direction? Yeah, I think, you know, I always say that my biggest thing is understand what has to flow, right? Begin to think of your systems in terms of the flow of information rather than in terms of the user interface talking to a database. You know, we've had this sort of big three-tier model kind of, okay, every problem is there's a database and there's a UI and the two talk to each other. Begin to think about, and a lot of, you know, enterprise architects already do this, but begin to think about the flow of information that you need from any event that occurs that you want to be able to respond to and get value from to any other, you know, any other point in your data, in your world where that has to link to. I would say that, you know, if you, rather than having the traditional object graph that you had before about your environment, if you had a data flow graph, a directed graph understating of your systems architecture in your company, it would be a really valuable and eye-opening thing to be able to take a look at. And then from there, you can begin to look at which of these data sets need to be a request response, which of these interactions or request responses should be API driven, which of these interactions should be about receiving information for as a stream from something else. And, you know, are there pieces of information where, you know what, like a lot of analytics systems, just, you know, have a data warehouse or a data lake and point your analytics systems at that. That's actually not a bad model in those situations. You don't have to process the stream directly to get your analytics unless you need real-time information about a specific aspect of what you're trying to deal with. Observability is a good one for that, you know, good understanding that, or if you're running campaigns in a retail environment and you want to understand how's that campaign going?

How are people responding to the campaign in real time? Those are the kinds of situations where you might want to process the stream right away. But in other situations, you're looking at what's the last month's, you know, report for X. In those situations, you may not need to do that. But as you get that understanding of how information flows and looking at it in terms of the flow of information to the business, then I think the next thing that you do is you begin to look at your application portfolio and understanding where would it make the most sense to refactor what we're doing from, you know, a call tree, like a services call tree to something that's more an event-driven architecture where maybe you're receiving the data and building your own kind of local representation of the state from that. And there's also systems that do, by the way, huge amounts of state at scale. Swim.ai is an interesting platform for that purpose where you could actually build a representational state model of a very complex system right there in memory on a surprisingly few number of systems from a scale perspective. And then take, you know, basically build either AI models or other kind of rule-based models that can respond to those state situations on an agent-by-agent basis. So each individual element can figure out how it responds to the world around it and take action. And so you can do, there's some really interesting new models that are out there that are worth looking at and seeing if they make your situation better. But in terms of your legacy infrastructure, you can also look at, you know, are there ways that I can take a data update and turn it into an event, right? Are there ways that I can, instead of doing a query on somebody else's database or calling a microservice even with a request response, is there a way that I could turn that around and have a microservice send me an event that I can just sit and listen to and respond to when it arrives? Those are the questions to begin to understand and begin to understand how to modify the flow of information in your world

so that it's as directed in the right direction as possible. And there's a minimum number of conversational elements that have to happen in order to enable it. So when I'm hearing this, two specific technologies come to mind. One is like serverless slash functions. And the other is Kafka or something, you know, along those lines. I think they call Kafka a message broker is probably unfair at this point. But, you know, and RabbitMQ or something, you know, along those lines is probably in there as well. So what are the roles of, you know, some of these technologies in this? I mean, is this where, you know, serverless or Kafka or anything like this becomes kind of a very commonplace? Yeah, well, I'm actually kind of writing a book on this topic right now. So if people can kind of look for that this fall, hopefully by the end of the year, it should be out. But there's a set of components that come together to kind of make the full end and piece of flow, right? The most important thing to kind of look at is right now, just making an integration work involves figuring out what somebody else's interface is for establishing the connection. What's the metadata they send on the stream that describes what the event is and the context of the event? What's the payload of the event and how to interpret the payload event? I think the first thing to understand is, you know, plan for a future in which that stuff becomes more and more commodity and standardized. Plan for a future in which what you want to do is make sure you understand your stream and you're looking for standards to consume in order to package events on that stream. There's a cloud event standard that CNCF has put out. That's a good thing to look at. Maybe that's the right option. MQTP, MQTT, these are all, you know, things, technologies that people who are kind of dealing with integration are probably somewhat aware of already. So you need that connection and you need that ability, those interfaces and protocols to make that connection work. And then you need, at the other end, you need processing capabilities. And that's where something like Kafka comes into play or swim.ai that I mentioned earlier

as well as a different version, a different kind of view of that. So Kafka has evolved from being strictly a queue, a log-based queue, and a collector of events and something that can handle high, high volumes of events coming in and putting them in a mechanism that you could use to read and understand those events to having some processing capability in addition to that, right? So it becomes more of a platform kind of view. But, and Rabbit as well is introducing log-based queuing as, in addition to message-based queuing. So queues are very important because you want to decouple the synchronicity. We talked about how important time is. You want to decouple that synchronicity between producer and consumer as much as possible. But there's other things too, right? There's the, you know, what are the sources? What are the devices? What are the, you know, where are you getting your events from? What are the technologies you're going to use to generate those events and package them correctly and put them out there and into the connection? And what are the syncs? What are the things that are going to be sort of the final resting places or final actions that an event goes through on any given branch that it takes? Because events at some point in time, the flow stops, right? And it becomes data at rest that can be analyzed later. And so there's all kinds of things there around, you know, if you look at just AWS's ecosystem alone, or if you look at even, you know, to a certain extent, some of the stuff that we've done with TAS and Tansu Application Service and others, there's a number of things you can kind of do with, you know, already today with having events come into being displayed in a visualization dashboard, for instance, or being used to trigger actions, both in a digital way or in a physical way. And, you know, you can have machines respond or robots respond to events as they come in. So those are all very possible. And you mentioned serverless. I think, you know, having really, really efficient stream-based processing is super important. And I, you know, I love Lambda as one aspect of that. I love something like step functions as one aspect of that.

And each of the cloud providers has their sort of versions of various aspects of that. But you have, you know, the ability to quickly take an action and make a decision or signal something else with a new event or whatever. Those things are going to be the nodes in the graph. So that's the really important part of it. Anyway, so that's what I say is the critical, are the critical elements to think of there. But there's a lot of, you know, I talk about a lot of different components that are really important. And the underlying, the cloud makes it possible in a lot of ways, because having a really reliable infrastructure that you know is going to run a certain way that's easily consumable at different scales is one of the enablers that makes this happen. And, and that's, that's both sort of, you know, public cloud, private cloud, you know, all those aspects, but be having that, not having firmly and fixed and coupled assigned hardware and, and network routes and things like that, but being able to reuse things much more easily is a big part of this as well. Yeah. And I mean, I just at a much higher level, I, I often go back to this idea of like, why, like, why are we talking about a lot of things now? And part of it is like the part of things, like we've been talking about certain things for years, but now we have the pieces in place to actually do it. Yeah. And it often starts with the cloud and starts with some of these data systems and the, all these, all these pieces. And then you can actually start to realize some of the, you know, maybe utopia and they're kind of, you know, big picture things that we've actually been talking about for a long time. So I think you're right. Like the cloud is an integral part. I'm just one more question, James, which is like, it seems like most or many people, many discussions come down to a matter of, you know, as we're talking about the technological stuff and the architectural aspects, it also comes into like the organizational part of things. And I remember we did a collection of blog posts with discover last year. And one part of it was the, the enterprise driven architecture that, you know, on the product side, but like behind the scenes, there was also some organizational changes and process changes and, and all these other quote unquote digital transformation activities going on.

So how do you, when you look at like adventure of an integration, just the adventure of an architecture at large, how do you think that plays into, you know, some of the discussions about organizational changes and processes and teams and, and what's the connection? Yeah. One of the things I think is really interesting about the last 10 years is the, the organizational models that we used when we were being very contextual in the way we built IT, meaning, you know, we had fixed resources, fixed ability to get things done. We had, you know, we had to be very, very careful in how we plan to use resources because they were incredibly expensive and incredibly difficult to even acquire and get installed. So we were very kind of construction oriented in the way we thought about software for a long time. And the last 15 years or so have really been about this shift to breaking out of that construction mold and getting more into what I call, you know, a more curated mode, right? More of like a gardener or more of a zookeeper in the sense that, you know, yeah, we have some fixed elements about what we're doing. We have some fixed constraints, but what we really want to do is help a system evolve that gives the kind of positive feedback loops that we want and eliminates as many of the negative feedback loops that we don't want. We want to shape and evolve and adapt as we go. And so the problems around event-driven architectures are very much like the problems around most modern application and, you know, cloud-native modern application development, whatever you want to call it, in that now really what you want is a much more flexible organization in terms of how you develop, but you want it to be built in an environment that can enable that flexibility and provide a contextual kind of platform. And so, you know, I always say it's really important if you have infrastructure you have to run, then absolutely you need an infrastructure team. Understand that their focus really is about delivering the capacity that you need to run software. Their job is to deliver the storage network and compute that you need in order to make a

software system run. So that's, that's, but that's super important. And so if you have data centers and most enterprises do that, it's critical to have a really good infrastructure team, making sure that they understand where the borderline is of what they're delivering. So the next element then is, you know, you have development teams and those are highly agile and you really have this problem today of how do you not organize in a way that your, your, your development teams are locked into certain organizations and certain activities. You want some ability, at least within a business unit, to be able to quickly adapt how resources are allocated to meet whatever the priorities are at the time. And so I think there's a lot of practices around pair programming and other things. It doesn't have to be pair programming, but pair programming helps with you being able to reassign people across teams a little bit more easily. You don't necessarily have to change their management, but you absolutely can change their assignments on a more regular basis and make it a little bit easier for them. And in the middle there, then, you know, what developers care about is getting their stuff into production. They don't care about finding capacity. That's the last thing they want to worry about. They just want to get their stuff in front of customers and start getting feedback from customers. And so from that perspective, then the last thing you need is a good path to production for developers. And to me, that's where the platform team fits in the middle, like firmly between helping understand how infrastructure gets consumed and working with the infrastructure team on making sure that software gets assigned to capacity in a, in a smart and intelligent way, but also at the same time, listening to developers and working with developers to eliminate as many bottlenecks in that process and many constraints in the development process as you can and working systematically at improving that development process over time as much as possible. And so I look at sort of that platform team element as being a really important organizational piece for providing standard building blocks like databases and messaging systems and, you know, all the Kafka stuff we talked about, all the event-driven pieces, all the underlying services

that enable an application team to build something that's business specific and flow information between themselves and other systems out there. And, you know, and also being able to provide the building blocks for actually writing code, providing the, you know, kind of trusted software supply chain that makes code easy to execute and makes code development easy to execute and getting to production something that security can go. Yeah, we're really comfortable if they build this certain way that what will come out the other end will pass on it. And so those things are really important and pieces that I think an organization, that's why an organization really needs to look at platforms and they really need to look at platform teams because even, you know, I, I ideally want them to use Tanzu application service because we think just makes it a billion times easier than doing just about anything else out there. But let's say that they don't want a platform like that. Well, at the very least, you need an organization that's worried about building the right path to production for your developers and simplifying, getting code to production as much as possible. And they could do it all with open source or all with something else, but it's still a need. And it's still something that I think, you know, we advise our executives on our exec customer executives on a regular basis that they really need to look at the role of the platform team as being that bridge from capacity to a path to production. All right. Insightful as always, James. Thanks a lot. I appreciate it, Derek. Good to talk to you, man. All right. There you have it. I hope you learned something. I certainly did. To follow James Urquhart, you can check him out, follow him on Twitter at James Urquhart. That's James spelled how you would normally spell James. And then U-R-Q-U-H-A-R-T. And to follow, to hear more about VMware's myriad toolings around cloud-native architectures and applications and platforms and event-driven architecture, just check out tanzu.vmware.com. Tanzu, T-A-N-Z-U.vmware.com.

Thanks for listening.

Tanzu Talk · 🍕 Get more pizza.