Hello, everyone. This is essentially part two of the previous episode, where we were talking with Matt Curry. But I thought we should split this one up into two, because it has a pretty much completely different topic that Kenny goes over. But with that, enjoy the episode. Well, on a related topic, in that it involves computers, what was the talk you gave about? So the talk that I gave was, I was presenting with Jakub Pilman. He lives in Poland, and he came all the way out here for our talk. And it was just about event sourcing, event-driven architectures. And we had a dialogue. So he built an application that was event-driven, and I just asked him questions. It was a lot of fun. So give us a definition of those two things, like what they mean nowadays. Definition of event sourcing? Yeah, event sourcing, what was the second one? It started with event-driven architectures. Yeah, yeah. So event-driven architectures are any architecture that really produces and consumes events to implement the business logic, right? Okay. Event sourcing itself is kind of an implementation detail. Once you have a stream of events that are ordered in time, you can use those events to reconstruct the state of a system. And that's important in distributed systems because so often you have inconsistencies in different applications. So if you have microservices that are communicating with each other to implement the business logic, you can have inconsistencies in the data. So that's why event-driven architectures are important. Right. And then so, like, I mean, I guess, have we talked about this before? Or maybe it was when you were on the Jeff and Dormain show when you talked about this. But anyways, like, I mean, essentially, it's like one of the main characteristics you start with is it's very asynchronous, right? Like, you don't have things processing in serial, and so you're just, like, firing off events. People listen for it. People. Parts of your system listen to it, and they decide to do something independently on their own. So everything is, in theory, very decoupled, right?
2:01Like, you don't have a lot of, like, blocking and waiting for stuff, and things just sort of are spread out there, and other things consume them and decide what they want to do, to repeat myself. Yeah. I think it just becomes easier for developers to be able to understand the behavior of systems when everything is modeled as events. Yeah. You know, way back when I, like, actually did programming, it seemed like, this is probably not the case, but the biggest, I mean, this is back in the early 2000s, but the biggest place where event-driven stuff would happen would be in GUIs, which... And that's still true today, right? Yeah. Which, I think, I mean, it's interesting, at least for me at the time, it was an instructive way of looking at it, because the nature of a GUI is, you don't want to, like, is it modal or whatever? You don't want to lock the whole GUI up when you do something. Well, there's a life cycle. Yeah, yeah. And as a part of that life cycle, it's driven by events. And so with functional programming, you're really subscribing to the type of events you want to listen for, and then you're implementing business logic in response to those events. Right. And so I guess it's a bit different from the event sourcing we're talking about today with microservices and distributed systems. It's more about the data, right? Once you take one database and you split it up into many databases, there's really no central management to make sure that that data is consistent across the architecture. Yeah, that's true. And hence, I'm not needing, like, the log so you can reconstruct everything and even all of that. Exactly. And that's how databases have always worked. So HA databases will have a transaction log. So every single operation that you commit to a database has an item in a list, essentially a log. And you can just replay that list in order to get the current state of the database. So now we're just moving that out across many separate applications. Right. You're just recreating a part of that domain wherever you need it. So then in this dialogue that you were having, like, what were you using to do all of your event sourcing and things like that? Like, did you have some example application and kind of, like, what did you walk through to demonstrate? Yeah, so Yakup implemented an event sourcing system from scratch using a hash map, which is in memory.
4:07And if you shut the application down, you're going to lose the data, right? But it shows you how to implement a system like this. And so let's say that you have credit cards. Now, each credit card has a unique ID. And you can index a set of events that are relevant to one card just by that ID. And so that's fundamental to how these event sourcing systems work. Now, if you wanted to save that data, you could put it in a database. But it would be much better if you had an API that was well-suited to that model. And so that's why Apache Kafka has become very popular is because it uses this idea of transaction law, which they call a topic. And each one of these topics might have, for one part of the domain, let's say a credit card again, this stream of events applies to all of the credit cards that you have. Oh, right, right, right. Yeah, and so you can recreate the state of any credit card, like the available balance or the limits, just by replaying those events. So you have the topic, which is basically like all of the events done on behalf of this entity, this credit card, if you will. And so if you're interested in like the life history of this credit card, you can sort of ask the topic. Like, tell me all the events related to this. So in a sense, it's like, what did they used to call it? It's like persistent. Like the events like don't just disappear once they get spread out. They're like stored somewhere. Right, right. It's kind of like graffiti, right? You have, maybe somebody comes along and they graffiti a tag a wall. And somebody else comes along and tags right over it. That's how systems are built without events, right? And so you can't really see what the history is. Now, there's many reasons why it's important and valuable. And mostly today, it's about data consistency. But also, as a side effect of having an event-driven system, you can understand the behavior of how people are using the application. And so you can have... How so? Like what does it give you that? How does it give you that? Because events model the behavior of the user, right? When a user takes an action, that produces an event.
6:08And so because you have all of the events in the entire system, you can join them together and create models that analyze behavior. So in theory, you can just replay everything that happens. Right, exactly. Which I guess in a non-event thing is a little difficult because usually, if you're doing... I don't know what the vocabulary is, but a non-event-driven thing where a transaction occurs, right? Where state changes and then you're done with it. You never really keep track of all the stuff that happened to get to that changed state. It's just like the state has now changed and I don't care how it happened. Versus an event thing, inherently, you have to keep track of the whole chain of stuff that happened. Yeah, you have a causal chain of events and you can understand and reason about what event caused what to happen. And so you understand cause and effect in that system, which allows you to better diagnose things, allows you to create real-time analytics on trends. So there's a whole set of side effects to what we're seeing today. Just to increase consistency of a system, we're also able now to analyze behavior. So do people do this kind of eventing stuff a lot or is it sort of like some wacky new thing from the past couple of years? People have been doing it for 30 years. Yeah, yeah. It's just becoming popular again today. I think it'll become popular again in 20 years. It's probably one of these things where we get to joke about small talk did it first. Yeah, I was going to ask you, like, how is it different than a lot of the messaging-based systems we've done in the past? Like, you know, TIBCO's out there. I think probably a lot of shops that have been running for a long time have, like, pretty big TIBCO installs. There are other messaging systems. Is it the ordering or, like, what is it that makes kind of an improvement in our architecture? It's all about cause and effect, right? I think very often we have problems running systems because we don't understand this chain of cause and effect of how you got to a certain state. Now, when something goes wrong, if you've just pasted over all of those events that caused that to happen, it becomes very difficult to diagnose root cause.
8:14So we've been doing this at different levels of abstraction for a while, but now we have these products like Apache Kafka that provides developers with an API that makes it easy to be able to store events in that model and be able to retrieve them. Yeah, I mean, my impression of, to reword the question, how it's different than an ESB, right, is that, one, like what you just said, is the way to interact with it is a lot more lightweight and easy. It's easier. There's less undifferentiated heavy lifting. Which I think, you know, using the pattern of your small top joke is like, you know, every five years we realize that the way we interact with our middleware five years ago was crap. Right, yeah. Like, we improve the interface for it, which is great, right? Like, that's what you want. And it also seems, and tell me if this is wrong, but it seems like with an ESB, you're not interested in kind of what you're implying, like the minutia of the cause and effect of what happened. You're more interested in like a gigantic payload in like a bus of just like, boom, here's the thing. It's more about, I feel like in an enterprise service bus driven thing, it's more about coordinating between big monolithic things and kind of just like passing stuff. It's more about integration than necessarily like, you hit on the asynchronous piece, right? Right, right. So it's like, it gives you asynchronous without much else, I think. Yeah. And the other thing that's interesting about ESB, which fits to Kenny's point, is like, there's a whole bunch of business logic that sits in that thing, which I think most people have been sort of saying all of the business logic actually sits in the producers and consumers. That's right, yeah. The old stupid network. So the events are just being delivered to whoever wants to listen to them, right? So if I wanted to listen to some stream of events, I can tap into that, and I'm not affecting anybody else by doing that, right? And I don't, it's self-service again. If it's self-service, it's easy to do. Developers are going to be able to do it without a problem. And I think that's where we're going today.
10:15And Apache Kafka allows that to happen. And is there something that, like, changed in the industry to have us, like, come readdress this topic? Like, was it the introduction and popularization of Kafka that really, like, pushed us in this direction? Or, like, is there something else going on with, like, the scale and, you know, the characteristics of the applications we're delivering that's kind of driving people to look at this as a pattern for future systems? It's the amount of work involved with setting up a system like this, right? If you want to think about going to production with a feature and you're thinking about how you're going to store the status of an account, developers aren't going to think about events with that use case, right? Because they have to get to production. And it's just too hard for them to worry about setting up a table that has, where each row represents an event for an account status, and then going back and querying that to get to the current state. Now, with Apache Kafka, they've added a key value store on top of it. They've bolted on a key value store. And so with this transaction log with Apache Kafka, as you're shipping these events every time a user takes an action, Apache Kafka can snapshot the current state of those objects and store that in the key value database. And that has made it even easier because now developers don't have to worry about setting up additional infrastructure. They can just use the API that's well-suited for that model and be able to use the key value store that's attached to Apache Kafka. Yeah. Yeah, it seems like, I mean, there's a continual, at least in my way of looking at it, push and pull, mostly pull, of like, to use all fancy words, like, deconstructing the relational database. But, like, essentially, like, we need a place that's secure and reliable to store our stuff. Yeah, right. Just like, we all can agree on that. And, like, for a long time, it's been, like, the relational database, right? And then, like, when you go over the explanation, what I hear is, like, well, now we've got this other, like, dealing with a relational database is not as awesome as, you get used to it after a while, but it's not super cool, right?
12:20Like, it would be better just to have a hash, right? Like, a map is cool. Like, a key value pair thing is conceptually simple and easy. Yeah, because the developers don't have to worry so much about changing schemas. Exactly. And then also you don't have to deal with the politics of the database, like, and all the people who manage it and all the, you know, hoity-toity-ness around it. You know, usually for better, but sometimes it's become for worse. And so then you get a good technology like Kafka come along, and assuming that, you know, it works. Yeah. And that it satisfies the number one requirement of any data-oriented thing is, like, don't lose data. Right. Like, then it's sort of like, well, we've got this better system that accomplishes the same things we need as a relational database, and then we don't have all the baggage that comes as a relational database. Exactly, yeah. And then you have to, like, reorient a little bit on, like, I imagine there's a lot of, like, philosophic talk in the event-driven board about, like, what even is a canonical representation of data, right? Right. Like, is there a single truth to anything? What's that even mean? Here's a 60-minute talk about it. But, you know, you sort of, like, you don't have that same assurance that in this one row that I got from a table, this is the order, right? Like, it's a little, it feels like it might be a little more relativistic. Yeah, well, I think you're spot on, yeah. This is a really, I'm glad that you brought this up because we're going to run into issues in the future with this model, and one of the issues that comes up today is the schema. Now, if you think about a chain of events that you've collected over 10 years, the schema for that event might change, right? As the business requirements change, maybe the first two years you had one schema and you didn't change it, and then the next two years you made a couple changes. Now, when you go to consume those events from the very beginning and that schema changes, if you're unaware of where it changed, it could cause an issue with recreating state. And so, that's one of the issues today that people are talking about when it comes to Apache Kafka. Yeah, and that raises, I mean, just to wrap it up, that raises another good point.
14:23Tell me if this is a real point, is that if you want to do things like going all the way back to your Flickr callback from way back when, like, if you want to be able to, like, almost effortlessly roll back changes, or do, like, not even A-B, but A-B-C-D-E-F-G, like, testing, that's probably too many things. But you want to, you know, run multiple versions of software in production to test it out, or for whatever reason, it seems like one of the major constraints you would always have is, like, not only multiple schemas in your database at the moment, but, like, with rolling back, like, if you have a major schema change, and then you have to run that and change the production database, and then you want to roll it back, I mean, you're kind of screwed. It's the rollbacks that get you. Yeah, yeah, whereas if you have the much more malleable, just key value store, it maybe is not a good idea to have all that cruft in there, but it's a lot easier to, like, not, what's the word? The word when, like, you can't go backward, right? Like, there's a certain point in data management where it's just, like, impossible to revert back to something, whereas it would seem like if you're dealing with a bunch of just key value things, you could have a scheme that you've got multiple schemas running that kind of work together. I don't know, it doesn't solve the problem, but I think it would be a lot easier than if part of my upgrade was going and changing and adding new tables and moving data around and stuff in relation to database. Right, right, yeah. So do the views actually solve this problem at all, Kenny? Like, you know, these data views that you can create? It does if you know in advance, right? So if you have one model that doesn't change, it's very easy, right? So if you talk about, let's say we have a stream of events, and at some interval there are changes to the schema. Well, if the team who is creating that view has snapshotted it throughout history and they've created these transformations as they've created these schema changes, then that model is going to be consistent in the future.
16:23But if you go back and try to rerun and create a different transformation of that model, you have to be aware of the schema changes. And so the snapshots are very valuable if you don't lose them. But once you do lose them, it's hard to recreate the state of that snapshot. Yeah. So you can have all of your 10 years of old data in different models in different schemas, and you could keep that. And as long as you're aware of what that representation is, it's easier to traipse between all that decade's worth of data. Yeah, yeah. And I think this is a solvable problem. So we'll probably see some changes to Apache Kafka that allows you to manage these schema changes. There's Apache Avro, which does focus on that. I don't know too much about it. I haven't really dug in too deep on it. But I know that it does handle schema evolution. Yeah. So I think it will be solved problem. To use my tired little saying, like back when I coded, I think eventually the database is what slowed our development ability down to a glacier pace. Not our development ability, but our ability to deliver new features. Because we would make all sorts of fun new things in code, and then you'd have to spend a couple of weeks writing the database upgrade scripts. And then, like I was talking about earlier, if those screw up, you're toast. Like there's no going back. What's even worse is that because you have deadlines, and you're not able to really do the right thing, you have to do something quick and dirty. And you have to make a trade-off. And you have to make a workaround. And that really muddies the water in terms of architecture. Yeah, and then it also gets to like the monolith decoupling promises of microservices, where if, and I remember things like this happening, where so let's say you've got a version 3.2. That you're deploying, which has 10 to 20 features in it. And so inevitably on Saturday night, someone runs the thing that updates the database, and then deploys the new code.
18:23And then you still have the old version running, and you do some quick smoke tests, and you switch over to like the new version. But what that means is that if something goes wrong, you lose out on deploying all those new features, right? Like you're truly coupled in that monolithic thing. And so until you fix that one little thing that screwed up the database, everyone else has to like wait to be deployed. Right, yeah. Whereas like slicing it, having more flexibility in the data layer and slicing it up like that means potentially you could have deployed those things, even though one of them was crap. Yeah, and now you have developers waiting instead of working on features. And I think this comes back to our discussion on DevOps, right? Where developers who are sitting idle because of something that they can't control is going to create distrust between factions. Yeah. All right, well, that's great. Well, we'll see everyone next time.