1
00:00:00,000 --> 00:00:05,300
Hello, everyone. This is essentially part two of the previous episode, where we were talking with Matt Curry.

2
00:00:05,740 --> 00:00:12,860
But I thought we should split this one up into two, because it has a pretty much completely different topic that Kenny goes over.

3
00:00:13,440 --> 00:00:15,560
But with that, enjoy the episode.

4
00:00:16,300 --> 00:00:25,060
Well, on a related topic, in that it involves computers, what was the talk you gave about?

5
00:00:26,020 --> 00:00:33,940
So the talk that I gave was, I was presenting with Jakub Pilman. He lives in Poland, and he came all the way out here for our talk.

6
00:00:34,160 --> 00:00:39,200
And it was just about event sourcing, event-driven architectures. And we had a dialogue.

7
00:00:39,760 --> 00:00:46,040
So he built an application that was event-driven, and I just asked him questions. It was a lot of fun.

8
00:00:46,400 --> 00:00:50,440
So give us a definition of those two things, like what they mean nowadays.

9
00:00:50,820 --> 00:00:51,940
Definition of event sourcing?

10
00:00:51,940 --> 00:00:55,520
Yeah, event sourcing, what was the second one? It started with event-driven architectures.

11
00:00:55,520 --> 00:00:55,880
Yeah, yeah.

12
00:00:55,880 --> 00:01:03,100
So event-driven architectures are any architecture that really produces and consumes events to implement the business logic, right?

13
00:01:03,340 --> 00:01:03,580
Okay.

14
00:01:04,300 --> 00:01:07,920
Event sourcing itself is kind of an implementation detail.

15
00:01:08,480 --> 00:01:14,840
Once you have a stream of events that are ordered in time, you can use those events to reconstruct the state of a system.

16
00:01:15,260 --> 00:01:21,360
And that's important in distributed systems because so often you have inconsistencies in different applications.

17
00:01:21,720 --> 00:01:28,760
So if you have microservices that are communicating with each other to implement the business logic, you can have inconsistencies in the data.

18
00:01:28,980 --> 00:01:30,860
So that's why event-driven architectures are important.

19
00:01:31,360 --> 00:01:31,580
Right.

20
00:01:31,940 --> 00:01:36,240
And then so, like, I mean, I guess, have we talked about this before?

21
00:01:36,400 --> 00:01:40,380
Or maybe it was when you were on the Jeff and Dormain show when you talked about this.

22
00:01:40,420 --> 00:01:46,760
But anyways, like, I mean, essentially, it's like one of the main characteristics you start with is it's very asynchronous, right?

23
00:01:46,760 --> 00:01:51,380
Like, you don't have things processing in serial, and so you're just, like, firing off events.

24
00:01:51,500 --> 00:01:52,520
People listen for it.

25
00:01:53,280 --> 00:01:53,560
People.

26
00:01:53,800 --> 00:01:57,480
Parts of your system listen to it, and they decide to do something independently on their own.

27
00:01:57,480 --> 00:02:01,280
So everything is, in theory, very decoupled, right?

28
00:02:01,360 --> 00:02:07,180
Like, you don't have a lot of, like, blocking and waiting for stuff, and things just sort of are spread out there,

29
00:02:07,300 --> 00:02:10,680
and other things consume them and decide what they want to do, to repeat myself.

30
00:02:10,960 --> 00:02:11,160
Yeah.

31
00:02:11,660 --> 00:02:17,920
I think it just becomes easier for developers to be able to understand the behavior of systems when everything is modeled as events.

32
00:02:18,600 --> 00:02:18,840
Yeah.

33
00:02:19,020 --> 00:02:25,020
You know, way back when I, like, actually did programming, it seemed like, this is probably not the case,

34
00:02:25,020 --> 00:02:32,460
but the biggest, I mean, this is back in the early 2000s, but the biggest place where event-driven stuff would happen would be in GUIs, which...

35
00:02:32,460 --> 00:02:34,080
And that's still true today, right?

36
00:02:34,220 --> 00:02:34,460
Yeah.

37
00:02:34,840 --> 00:02:38,900
Which, I think, I mean, it's interesting, at least for me at the time, it was an instructive way of looking at it,

38
00:02:38,920 --> 00:02:44,420
because the nature of a GUI is, you don't want to, like, is it modal or whatever?

39
00:02:44,460 --> 00:02:46,960
You don't want to lock the whole GUI up when you do something.

40
00:02:47,180 --> 00:02:48,100
Well, there's a life cycle.

41
00:02:48,320 --> 00:02:48,820
Yeah, yeah.

42
00:02:48,820 --> 00:02:51,820
And as a part of that life cycle, it's driven by events.

43
00:02:52,420 --> 00:02:57,840
And so with functional programming, you're really subscribing to the type of events you want to listen for,

44
00:02:58,380 --> 00:03:01,720
and then you're implementing business logic in response to those events.

45
00:03:02,200 --> 00:03:02,440
Right.

46
00:03:02,640 --> 00:03:08,360
And so I guess it's a bit different from the event sourcing we're talking about today with microservices and distributed systems.

47
00:03:08,740 --> 00:03:10,500
It's more about the data, right?

48
00:03:10,760 --> 00:03:13,800
Once you take one database and you split it up into many databases,

49
00:03:14,500 --> 00:03:19,040
there's really no central management to make sure that that data is consistent across the architecture.

50
00:03:19,360 --> 00:03:19,780
Yeah, that's true.

51
00:03:19,860 --> 00:03:23,900
And hence, I'm not needing, like, the log so you can reconstruct everything and even all of that.

52
00:03:24,040 --> 00:03:24,440
Exactly.

53
00:03:24,580 --> 00:03:26,500
And that's how databases have always worked.

54
00:03:26,680 --> 00:03:29,220
So HA databases will have a transaction log.

55
00:03:29,600 --> 00:03:36,260
So every single operation that you commit to a database has an item in a list, essentially a log.

56
00:03:36,820 --> 00:03:40,520
And you can just replay that list in order to get the current state of the database.

57
00:03:41,020 --> 00:03:44,280
So now we're just moving that out across many separate applications.

58
00:03:44,780 --> 00:03:44,920
Right.

59
00:03:45,040 --> 00:03:48,420
You're just recreating a part of that domain wherever you need it.

60
00:03:48,720 --> 00:03:55,640
So then in this dialogue that you were having, like, what were you using to do all of your event sourcing and things like that?

61
00:03:55,800 --> 00:04:00,600
Like, did you have some example application and kind of, like, what did you walk through to demonstrate?

62
00:04:00,980 --> 00:04:07,420
Yeah, so Yakup implemented an event sourcing system from scratch using a hash map, which is in memory.

63
00:04:07,800 --> 00:04:10,380
And if you shut the application down, you're going to lose the data, right?

64
00:04:10,820 --> 00:04:13,200
But it shows you how to implement a system like this.

65
00:04:13,460 --> 00:04:15,240
And so let's say that you have credit cards.

66
00:04:15,620 --> 00:04:17,700
Now, each credit card has a unique ID.

67
00:04:18,240 --> 00:04:23,000
And you can index a set of events that are relevant to one card just by that ID.

68
00:04:23,560 --> 00:04:27,840
And so that's fundamental to how these event sourcing systems work.

69
00:04:28,720 --> 00:04:32,180
Now, if you wanted to save that data, you could put it in a database.

70
00:04:32,700 --> 00:04:37,160
But it would be much better if you had an API that was well-suited to that model.

71
00:04:37,760 --> 00:04:44,660
And so that's why Apache Kafka has become very popular is because it uses this idea of transaction law, which they call a topic.

72
00:04:45,200 --> 00:04:50,120
And each one of these topics might have, for one part of the domain, let's say a credit card again,

73
00:04:50,660 --> 00:04:54,840
this stream of events applies to all of the credit cards that you have.

74
00:04:55,880 --> 00:04:56,540
Oh, right, right, right.

75
00:04:56,540 --> 00:05:04,460
Yeah, and so you can recreate the state of any credit card, like the available balance or the limits, just by replaying those events.

76
00:05:04,640 --> 00:05:12,220
So you have the topic, which is basically like all of the events done on behalf of this entity, this credit card, if you will.

77
00:05:12,300 --> 00:05:18,320
And so if you're interested in like the life history of this credit card, you can sort of ask the topic.

78
00:05:18,680 --> 00:05:20,840
Like, tell me all the events related to this.

79
00:05:20,920 --> 00:05:24,200
So in a sense, it's like, what did they used to call it?

80
00:05:24,220 --> 00:05:24,920
It's like persistent.

81
00:05:25,300 --> 00:05:28,540
Like the events like don't just disappear once they get spread out.

82
00:05:28,680 --> 00:05:29,640
They're like stored somewhere.

83
00:05:29,960 --> 00:05:30,480
Right, right.

84
00:05:30,820 --> 00:05:32,820
It's kind of like graffiti, right?

85
00:05:33,520 --> 00:05:37,060
You have, maybe somebody comes along and they graffiti a tag a wall.

86
00:05:37,320 --> 00:05:39,800
And somebody else comes along and tags right over it.

87
00:05:39,920 --> 00:05:43,040
That's how systems are built without events, right?

88
00:05:43,300 --> 00:05:45,460
And so you can't really see what the history is.

89
00:05:46,000 --> 00:05:49,120
Now, there's many reasons why it's important and valuable.

90
00:05:49,620 --> 00:05:52,020
And mostly today, it's about data consistency.

91
00:05:52,580 --> 00:05:58,800
But also, as a side effect of having an event-driven system, you can understand the behavior of how people are using the application.

92
00:05:59,420 --> 00:06:00,160
And so you can have...

93
00:06:00,160 --> 00:06:00,440
How so?

94
00:06:00,560 --> 00:06:01,960
Like what does it give you that?

95
00:06:02,060 --> 00:06:02,660
How does it give you that?

96
00:06:02,800 --> 00:06:04,880
Because events model the behavior of the user, right?

97
00:06:04,900 --> 00:06:07,880
When a user takes an action, that produces an event.

98
00:06:08,240 --> 00:06:16,340
And so because you have all of the events in the entire system, you can join them together and create models that analyze behavior.

99
00:06:16,380 --> 00:06:19,020
So in theory, you can just replay everything that happens.

100
00:06:19,200 --> 00:06:19,660
Right, exactly.

101
00:06:19,780 --> 00:06:24,400
Which I guess in a non-event thing is a little difficult because usually, if you're doing...

102
00:06:24,400 --> 00:06:28,800
I don't know what the vocabulary is, but a non-event-driven thing where a transaction occurs, right?

103
00:06:28,800 --> 00:06:31,160
Where state changes and then you're done with it.

104
00:06:31,660 --> 00:06:35,640
You never really keep track of all the stuff that happened to get to that changed state.

105
00:06:35,820 --> 00:06:38,180
It's just like the state has now changed and I don't care how it happened.

106
00:06:38,260 --> 00:06:43,840
Versus an event thing, inherently, you have to keep track of the whole chain of stuff that happened.

107
00:06:44,000 --> 00:06:50,220
Yeah, you have a causal chain of events and you can understand and reason about what event caused what to happen.

108
00:06:50,220 --> 00:06:59,180
And so you understand cause and effect in that system, which allows you to better diagnose things, allows you to create real-time analytics on trends.

109
00:06:59,720 --> 00:07:03,800
So there's a whole set of side effects to what we're seeing today.

110
00:07:03,940 --> 00:07:08,660
Just to increase consistency of a system, we're also able now to analyze behavior.

111
00:07:08,660 --> 00:07:15,240
So do people do this kind of eventing stuff a lot or is it sort of like some wacky new thing from the past couple of years?

112
00:07:15,400 --> 00:07:17,180
People have been doing it for 30 years.

113
00:07:17,480 --> 00:07:17,760
Yeah, yeah.

114
00:07:17,780 --> 00:07:19,860
It's just becoming popular again today.

115
00:07:20,300 --> 00:07:22,420
I think it'll become popular again in 20 years.

116
00:07:22,740 --> 00:07:25,700
It's probably one of these things where we get to joke about small talk did it first.

117
00:07:26,100 --> 00:07:35,740
Yeah, I was going to ask you, like, how is it different than a lot of the messaging-based systems we've done in the past?

118
00:07:36,000 --> 00:07:37,400
Like, you know, TIBCO's out there.

119
00:07:37,700 --> 00:07:43,740
I think probably a lot of shops that have been running for a long time have, like, pretty big TIBCO installs.

120
00:07:44,160 --> 00:07:46,660
There are other messaging systems.

121
00:07:46,880 --> 00:07:54,200
Is it the ordering or, like, what is it that makes kind of an improvement in our architecture?

122
00:07:54,700 --> 00:07:56,620
It's all about cause and effect, right?

123
00:07:56,800 --> 00:08:05,980
I think very often we have problems running systems because we don't understand this chain of cause and effect of how you got to a certain state.

124
00:08:06,260 --> 00:08:14,120
Now, when something goes wrong, if you've just pasted over all of those events that caused that to happen, it becomes very difficult to diagnose root cause.

125
00:08:14,620 --> 00:08:29,120
So we've been doing this at different levels of abstraction for a while, but now we have these products like Apache Kafka that provides developers with an API that makes it easy to be able to store events in that model and be able to retrieve them.

126
00:08:29,880 --> 00:08:43,840
Yeah, I mean, my impression of, to reword the question, how it's different than an ESB, right, is that, one, like what you just said, is the way to interact with it is a lot more lightweight and easy.

127
00:08:44,220 --> 00:08:44,740
It's easier.

128
00:08:44,840 --> 00:08:46,640
There's less undifferentiated heavy lifting.

129
00:08:46,660 --> 00:08:55,020
Which I think, you know, using the pattern of your small top joke is like, you know, every five years we realize that the way we interact with our middleware five years ago was crap.

130
00:08:55,420 --> 00:08:55,960
Right, yeah.

131
00:08:55,960 --> 00:08:58,420
Like, we improve the interface for it, which is great, right?

132
00:08:58,460 --> 00:08:59,240
Like, that's what you want.

133
00:08:59,460 --> 00:09:10,720
And it also seems, and tell me if this is wrong, but it seems like with an ESB, you're not interested in kind of what you're implying, like the minutia of the cause and effect of what happened.

134
00:09:10,800 --> 00:09:16,120
You're more interested in like a gigantic payload in like a bus of just like, boom, here's the thing.

135
00:09:16,200 --> 00:09:26,920
It's more about, I feel like in an enterprise service bus driven thing, it's more about coordinating between big monolithic things and kind of just like passing stuff.

136
00:09:26,920 --> 00:09:32,500
It's more about integration than necessarily like, you hit on the asynchronous piece, right?

137
00:09:32,640 --> 00:09:32,960
Right, right.

138
00:09:32,960 --> 00:09:36,960
So it's like, it gives you asynchronous without much else, I think.

139
00:09:37,260 --> 00:09:37,440
Yeah.

140
00:09:37,480 --> 00:09:53,380
And the other thing that's interesting about ESB, which fits to Kenny's point, is like, there's a whole bunch of business logic that sits in that thing, which I think most people have been sort of saying all of the business logic actually sits in the producers and consumers.

141
00:09:53,560 --> 00:09:54,560
That's right, yeah.

142
00:09:54,900 --> 00:09:56,060
The old stupid network.

143
00:09:56,740 --> 00:10:00,720
So the events are just being delivered to whoever wants to listen to them, right?

144
00:10:00,860 --> 00:10:07,400
So if I wanted to listen to some stream of events, I can tap into that, and I'm not affecting anybody else by doing that, right?

145
00:10:07,560 --> 00:10:09,080
And I don't, it's self-service again.

146
00:10:09,260 --> 00:10:10,780
If it's self-service, it's easy to do.

147
00:10:11,220 --> 00:10:13,000
Developers are going to be able to do it without a problem.

148
00:10:13,460 --> 00:10:15,080
And I think that's where we're going today.

149
00:10:15,280 --> 00:10:17,580
And Apache Kafka allows that to happen.

150
00:10:17,940 --> 00:10:25,720
And is there something that, like, changed in the industry to have us, like, come readdress this topic?

151
00:10:25,940 --> 00:10:31,540
Like, was it the introduction and popularization of Kafka that really, like, pushed us in this direction?

152
00:10:31,760 --> 00:10:44,700
Or, like, is there something else going on with, like, the scale and, you know, the characteristics of the applications we're delivering that's kind of driving people to look at this as a pattern for future systems?

153
00:10:45,280 --> 00:10:48,900
It's the amount of work involved with setting up a system like this, right?

154
00:10:48,940 --> 00:11:00,740
If you want to think about going to production with a feature and you're thinking about how you're going to store the status of an account, developers aren't going to think about events with that use case, right?

155
00:11:00,760 --> 00:11:01,700
Because they have to get to production.

156
00:11:01,860 --> 00:11:11,820
And it's just too hard for them to worry about setting up a table that has, where each row represents an event for an account status, and then going back and querying that to get to the current state.

157
00:11:12,160 --> 00:11:17,760
Now, with Apache Kafka, they've added a key value store on top of it.

158
00:11:18,000 --> 00:11:19,540
They've bolted on a key value store.

159
00:11:20,100 --> 00:11:31,500
And so with this transaction log with Apache Kafka, as you're shipping these events every time a user takes an action, Apache Kafka can snapshot the current state of those objects and store that in the key value database.

160
00:11:32,240 --> 00:11:37,940
And that has made it even easier because now developers don't have to worry about setting up additional infrastructure.

161
00:11:38,080 --> 00:11:46,060
They can just use the API that's well-suited for that model and be able to use the key value store that's attached to Apache Kafka.

162
00:11:46,720 --> 00:11:46,920
Yeah.

163
00:11:47,180 --> 00:11:58,320
Yeah, it seems like, I mean, there's a continual, at least in my way of looking at it, push and pull, mostly pull, of like, to use all fancy words, like, deconstructing the relational database.

164
00:11:58,880 --> 00:12:03,980
But, like, essentially, like, we need a place that's secure and reliable to store our stuff.

165
00:12:04,540 --> 00:12:04,720
Yeah, right.

166
00:12:04,720 --> 00:12:06,680
Just like, we all can agree on that.

167
00:12:07,080 --> 00:12:10,640
And, like, for a long time, it's been, like, the relational database, right?

168
00:12:10,680 --> 00:12:20,920
And then, like, when you go over the explanation, what I hear is, like, well, now we've got this other, like, dealing with a relational database is not as awesome as, you get used to it after a while, but it's not super cool, right?

169
00:12:20,920 --> 00:12:23,320
Like, it would be better just to have a hash, right?

170
00:12:23,420 --> 00:12:25,100
Like, a map is cool.

171
00:12:25,240 --> 00:12:28,140
Like, a key value pair thing is conceptually simple and easy.

172
00:12:29,000 --> 00:12:32,880
Yeah, because the developers don't have to worry so much about changing schemas.

173
00:12:32,980 --> 00:12:33,320
Exactly.

174
00:12:33,640 --> 00:12:40,000
And then also you don't have to deal with the politics of the database, like, and all the people who manage it and all the, you know, hoity-toity-ness around it.

175
00:12:40,080 --> 00:12:43,460
You know, usually for better, but sometimes it's become for worse.

176
00:12:43,560 --> 00:12:49,480
And so then you get a good technology like Kafka come along, and assuming that, you know, it works.

177
00:12:50,200 --> 00:12:50,560
Yeah.

178
00:12:50,780 --> 00:12:54,880
And that it satisfies the number one requirement of any data-oriented thing is, like, don't lose data.

179
00:12:55,300 --> 00:12:55,700
Right.

180
00:12:56,940 --> 00:13:06,800
Like, then it's sort of like, well, we've got this better system that accomplishes the same things we need as a relational database, and then we don't have all the baggage that comes as a relational database.

181
00:13:07,340 --> 00:13:07,860
Exactly, yeah.

182
00:13:07,860 --> 00:13:18,460
And then you have to, like, reorient a little bit on, like, I imagine there's a lot of, like, philosophic talk in the event-driven board about, like, what even is a canonical representation of data, right?

183
00:13:18,460 --> 00:13:18,540
Right.

184
00:13:18,540 --> 00:13:20,540
Like, is there a single truth to anything?

185
00:13:20,760 --> 00:13:21,480
What's that even mean?

186
00:13:21,540 --> 00:13:22,860
Here's a 60-minute talk about it.

187
00:13:23,020 --> 00:13:31,820
But, you know, you sort of, like, you don't have that same assurance that in this one row that I got from a table, this is the order, right?

188
00:13:31,920 --> 00:13:37,640
Like, it's a little, it feels like it might be a little more relativistic.

189
00:13:37,880 --> 00:13:39,240
Yeah, well, I think you're spot on, yeah.

190
00:13:39,340 --> 00:13:47,840
This is a really, I'm glad that you brought this up because we're going to run into issues in the future with this model, and one of the issues that comes up today is the schema.

191
00:13:48,280 --> 00:13:56,040
Now, if you think about a chain of events that you've collected over 10 years, the schema for that event might change, right?

192
00:13:56,140 --> 00:14:03,980
As the business requirements change, maybe the first two years you had one schema and you didn't change it, and then the next two years you made a couple changes.

193
00:14:04,660 --> 00:14:13,720
Now, when you go to consume those events from the very beginning and that schema changes, if you're unaware of where it changed, it could cause an issue with recreating state.

194
00:14:14,120 --> 00:14:18,960
And so, that's one of the issues today that people are talking about when it comes to Apache Kafka.

195
00:14:19,240 --> 00:14:23,040
Yeah, and that raises, I mean, just to wrap it up, that raises another good point.

196
00:14:23,040 --> 00:14:33,060
Tell me if this is a real point, is that if you want to do things like going all the way back to your Flickr callback from way back when,

197
00:14:33,480 --> 00:14:43,840
like, if you want to be able to, like, almost effortlessly roll back changes, or do, like, not even A-B, but A-B-C-D-E-F-G, like, testing, that's probably too many things.

198
00:14:44,120 --> 00:14:49,740
But you want to, you know, run multiple versions of software in production to test it out, or for whatever reason,

199
00:14:50,340 --> 00:14:56,900
it seems like one of the major constraints you would always have is, like, not only multiple schemas in your database at the moment,

200
00:14:57,460 --> 00:15:03,520
but, like, with rolling back, like, if you have a major schema change, and then you have to run that and change the production database,

201
00:15:03,740 --> 00:15:06,000
and then you want to roll it back, I mean, you're kind of screwed.

202
00:15:06,240 --> 00:15:07,420
It's the rollbacks that get you.

203
00:15:07,420 --> 00:15:12,580
Yeah, yeah, whereas if you have the much more malleable, just key value store,

204
00:15:13,080 --> 00:15:19,980
it maybe is not a good idea to have all that cruft in there, but it's a lot easier to, like, not, what's the word?

205
00:15:20,820 --> 00:15:23,000
The word when, like, you can't go backward, right?

206
00:15:23,080 --> 00:15:28,200
Like, there's a certain point in data management where it's just, like, impossible to revert back to something,

207
00:15:28,300 --> 00:15:32,020
whereas it would seem like if you're dealing with a bunch of just key value things,

208
00:15:32,100 --> 00:15:37,280
you could have a scheme that you've got multiple schemas running that kind of work together.

209
00:15:37,440 --> 00:15:40,680
I don't know, it doesn't solve the problem, but I think it would be a lot easier

210
00:15:40,680 --> 00:15:47,640
than if part of my upgrade was going and changing and adding new tables and moving data around and stuff in relation to database.

211
00:15:48,120 --> 00:15:49,120
Right, right, yeah.

212
00:15:49,600 --> 00:15:53,360
So do the views actually solve this problem at all, Kenny?

213
00:15:53,560 --> 00:15:56,720
Like, you know, these data views that you can create?

214
00:15:56,720 --> 00:15:59,120
It does if you know in advance, right?

215
00:15:59,260 --> 00:16:03,180
So if you have one model that doesn't change, it's very easy, right?

216
00:16:03,280 --> 00:16:05,660
So if you talk about, let's say we have a stream of events,

217
00:16:06,200 --> 00:16:09,780
and at some interval there are changes to the schema.

218
00:16:10,500 --> 00:16:16,540
Well, if the team who is creating that view has snapshotted it throughout history

219
00:16:16,540 --> 00:16:21,220
and they've created these transformations as they've created these schema changes,

220
00:16:21,660 --> 00:16:23,680
then that model is going to be consistent in the future.

221
00:16:23,680 --> 00:16:27,740
But if you go back and try to rerun and create a different transformation of that model,

222
00:16:27,900 --> 00:16:29,900
you have to be aware of the schema changes.

223
00:16:30,340 --> 00:16:33,560
And so the snapshots are very valuable if you don't lose them.

224
00:16:33,980 --> 00:16:37,120
But once you do lose them, it's hard to recreate the state of that snapshot.

225
00:16:37,540 --> 00:16:37,700
Yeah.

226
00:16:38,220 --> 00:16:45,800
So you can have all of your 10 years of old data in different models in different schemas,

227
00:16:45,920 --> 00:16:47,180
and you could keep that.

228
00:16:47,700 --> 00:16:50,020
And as long as you're aware of what that representation is,

229
00:16:50,060 --> 00:16:53,280
it's easier to traipse between all that decade's worth of data.

230
00:16:53,500 --> 00:16:54,000
Yeah, yeah.

231
00:16:54,280 --> 00:16:55,700
And I think this is a solvable problem.

232
00:16:55,940 --> 00:17:02,480
So we'll probably see some changes to Apache Kafka that allows you to manage these schema changes.

233
00:17:03,000 --> 00:17:06,080
There's Apache Avro, which does focus on that.

234
00:17:06,280 --> 00:17:07,440
I don't know too much about it.

235
00:17:07,620 --> 00:17:09,520
I haven't really dug in too deep on it.

236
00:17:09,780 --> 00:17:12,300
But I know that it does handle schema evolution.

237
00:17:12,520 --> 00:17:12,680
Yeah.

238
00:17:12,920 --> 00:17:14,320
So I think it will be solved problem.

239
00:17:14,340 --> 00:17:16,780
To use my tired little saying, like back when I coded,

240
00:17:17,260 --> 00:17:26,240
I think eventually the database is what slowed our development ability down to a glacier pace.

241
00:17:26,540 --> 00:17:29,600
Not our development ability, but our ability to deliver new features.

242
00:17:29,880 --> 00:17:33,200
Because we would make all sorts of fun new things in code,

243
00:17:33,200 --> 00:17:36,900
and then you'd have to spend a couple of weeks writing the database upgrade scripts.

244
00:17:37,420 --> 00:17:41,140
And then, like I was talking about earlier, if those screw up, you're toast.

245
00:17:41,880 --> 00:17:44,420
Like there's no going back.

246
00:17:44,520 --> 00:17:46,900
What's even worse is that because you have deadlines,

247
00:17:47,160 --> 00:17:48,960
and you're not able to really do the right thing,

248
00:17:49,560 --> 00:17:51,520
you have to do something quick and dirty.

249
00:17:51,900 --> 00:17:53,300
And you have to make a trade-off.

250
00:17:53,380 --> 00:17:54,320
And you have to make a workaround.

251
00:17:54,460 --> 00:17:56,960
And that really muddies the water in terms of architecture.

252
00:17:57,740 --> 00:18:02,560
Yeah, and then it also gets to like the monolith decoupling promises of microservices,

253
00:18:02,960 --> 00:18:06,780
where if, and I remember things like this happening,

254
00:18:06,960 --> 00:18:11,600
where so let's say you've got a version 3.2.

255
00:18:11,880 --> 00:18:14,740
That you're deploying, which has 10 to 20 features in it.

256
00:18:15,660 --> 00:18:18,440
And so inevitably on Saturday night,

257
00:18:19,160 --> 00:18:21,620
someone runs the thing that updates the database,

258
00:18:21,940 --> 00:18:23,300
and then deploys the new code.

259
00:18:23,880 --> 00:18:25,480
And then you still have the old version running,

260
00:18:25,680 --> 00:18:27,000
and you do some quick smoke tests,

261
00:18:27,040 --> 00:18:29,480
and you switch over to like the new version.

262
00:18:30,480 --> 00:18:32,920
But what that means is that if something goes wrong,

263
00:18:33,120 --> 00:18:35,960
you lose out on deploying all those new features, right?

264
00:18:36,020 --> 00:18:38,400
Like you're truly coupled in that monolithic thing.

265
00:18:38,480 --> 00:18:41,300
And so until you fix that one little thing that screwed up the database,

266
00:18:41,760 --> 00:18:44,140
everyone else has to like wait to be deployed.

267
00:18:44,300 --> 00:18:44,780
Right, yeah.

268
00:18:44,780 --> 00:18:45,740
Whereas like slicing it,

269
00:18:46,180 --> 00:18:49,120
having more flexibility in the data layer and slicing it up like that

270
00:18:49,120 --> 00:18:51,200
means potentially you could have deployed those things,

271
00:18:51,260 --> 00:18:52,680
even though one of them was crap.

272
00:18:52,900 --> 00:18:56,320
Yeah, and now you have developers waiting instead of working on features.

273
00:18:56,800 --> 00:19:00,660
And I think this comes back to our discussion on DevOps, right?

274
00:19:01,300 --> 00:19:03,400
Where developers who are sitting idle

275
00:19:03,400 --> 00:19:05,460
because of something that they can't control

276
00:19:05,460 --> 00:19:07,360
is going to create distrust between factions.

277
00:19:07,960 --> 00:19:08,140
Yeah.

278
00:19:08,580 --> 00:19:09,660
All right, well, that's great.

279
00:19:09,880 --> 00:19:11,960
Well, we'll see everyone next time.