1
00:00:00,000 --> 00:00:04,160
Well, here we are at SpringOne Platform, the most awesome named conference of the year.

2
00:00:04,340 --> 00:00:04,860
Don't you agree?

3
00:00:05,060 --> 00:00:05,620
Of course.

4
00:00:05,800 --> 00:00:08,580
And I have a guest with myself. You want to introduce yourself?

5
00:00:08,760 --> 00:00:11,200
Yeah. Hi, everybody. My name is Marcin Grzejszczak.

6
00:00:11,480 --> 00:00:15,360
I'm sorry for my surname, which is impossible to pronounce by foreigners.

7
00:00:16,140 --> 00:00:21,420
I'm working for the Spring Cloud team on basically mostly Sleuth,

8
00:00:21,980 --> 00:00:30,040
which is our implementation of distributed tracing with Spring Cloud and Spring Boot.

9
00:00:30,300 --> 00:00:34,900
And the other project is Spring Cloud Contract, but we're going to focus on the first one, I guess.

10
00:00:34,900 --> 00:00:36,560
Oh, yeah. I forgot about that one, too.

11
00:00:36,700 --> 00:00:42,460
We were walking between, I don't know, some meetup talk or something in a bar in a park,

12
00:00:42,640 --> 00:00:46,000
and you were telling me all about that last one. Now I remember.

13
00:00:46,360 --> 00:00:46,500
Yeah.

14
00:00:47,060 --> 00:00:48,660
Yeah, a bar in a park. Very nice.

15
00:00:48,780 --> 00:00:53,280
I mean, that's typical stuff that you do in a bar in a park and talk about Spring Cloud.

16
00:00:53,960 --> 00:00:58,020
Exactly. So how long have you been working on this Sleuth stuff?

17
00:00:58,380 --> 00:01:02,500
Actually, I joined Pivotal in December. So since December.

18
00:01:02,500 --> 00:01:05,860
Yeah, yeah, yeah. And what made you interested in it?

19
00:01:06,260 --> 00:01:10,040
Actually, I started in my previous company.

20
00:01:11,020 --> 00:01:15,380
We started, I didn't even know that it's called distributed tracing back then,

21
00:01:15,820 --> 00:01:23,280
but we have implemented in our open source solution that was related to microservices based on Groovy.

22
00:01:23,820 --> 00:01:26,240
We created correlation ID passing.

23
00:01:26,420 --> 00:01:34,660
So we did log correlation by ourselves, and I've contributed all of that to Spring Cloud.

24
00:01:35,080 --> 00:01:35,580
All right.

25
00:01:35,580 --> 00:01:37,220
So you got in through code.

26
00:01:37,460 --> 00:01:42,860
Yes. So before even joining Pivotal, I've already contributed some stuff to Spring Cloud Sleuth.

27
00:01:43,300 --> 00:01:50,920
And so let's start talking about Sleuth and Zipkin underneath it by saying the problem you were trying to solve.

28
00:01:51,140 --> 00:01:55,040
So what was the issue that you had that brought you to doing this?

29
00:01:55,140 --> 00:01:58,740
And are there things that you tried that you decided not to go with instead?

30
00:01:59,080 --> 00:02:00,500
Or like what brought you to it all?

31
00:02:00,640 --> 00:02:06,240
So basically, we had a problem that, I mean, we had a distributed system.

32
00:02:08,000 --> 00:02:15,220
We've invested a lot of time to, let's say, provide some standards of logging and running applications.

33
00:02:15,220 --> 00:02:18,920
Some basic stuff that you do when you want to go in the microservice direction.

34
00:02:19,920 --> 00:02:36,040
And the problem was that when an exception was thrown somewhere in the system, we couldn't correlate that with any sort of, like, user who clicked around or with some sort of a situation that happened in the system.

35
00:02:36,360 --> 00:02:41,840
I mean, there was an exception thrown somewhere, and then we didn't know what happened before and what happened after.

36
00:02:42,160 --> 00:02:42,580
Oh, right.

37
00:02:42,700 --> 00:02:44,900
So it was like going into a dark room to try to find something.

38
00:02:44,900 --> 00:02:49,060
So what we wanted to have is, like, you know, some breadcrumbs that we could follow, right?

39
00:02:49,060 --> 00:02:53,500
It happened here, then something happened, and so on and so forth.

40
00:02:53,920 --> 00:03:09,520
At that time, our correlation ID, it was actually even, it was not, let's say, proper correlation ID in terms of Zipkin, but our correlation ID had a user ID attached.

41
00:03:09,520 --> 00:03:25,380
So when a user, like somebody from customer service was clicking around, he had an exception, he called us, we knew who he is or she, so we could correlate easily who clicked, then we knew when the exception happened.

42
00:03:25,380 --> 00:03:41,280
So then we could find the proper correlation ID, and we went to some, at that time, we had Elasticsearch.org slash Kibana stack, so we went to Kibana, and we could see all the, you know, historical requests before the exception and what happened after.

43
00:03:41,440 --> 00:03:47,800
So that way, in a distributed system where we had quite a few microservices, a couple of monoliths, we could find what happened.

44
00:03:48,300 --> 00:03:58,360
Right, so you have a user, like, clicking around doing something, it's always a bad word because it's overloaded, but they're doing some sort of transaction or some action that begins and ends.

45
00:03:58,420 --> 00:03:58,660
Yes.

46
00:03:58,800 --> 00:04:10,500
And spans a lot of different services, and so something will go wrong, and I guess, you know, you can look at the logs and, like, try to figure out, like, which logs are related to that user by time and machine they were on.

47
00:04:10,680 --> 00:04:26,620
But then, as you're getting to, like, if you can somehow pass some unique ID through to every component of that, that gets you a little better as far as filtering the logs out to know that it's related to not only this user and not only this part in time, but this intention of them doing something, this transaction.

48
00:04:26,760 --> 00:04:36,780
Yeah, precisely, so in the distributed tracing nomenclature, this correlation ID I was talking about, and what you have just mentioned, it's called trace ID.

49
00:04:36,780 --> 00:04:50,700
So, yeah, for a single, let's say, business operation that can span throughout, you know, different microservices or monoliths, you have always the same value of that ID.

50
00:04:51,900 --> 00:05:01,040
Coming back to what you said, actually, one of the architects back in that company had an idea because I said that, I mean, right now we are blind.

51
00:05:01,340 --> 00:05:04,980
We cannot see what's happening in the system for a particular problem.

52
00:05:05,600 --> 00:05:12,640
So, he suggested that what we can do is to SSH to every machine and tail and grab the logs, right?

53
00:05:13,680 --> 00:05:15,960
So, tail the logs and grab for an error.

54
00:05:16,860 --> 00:05:21,520
So, of course, you can do that, but I don't think it's the best solution.

55
00:05:22,280 --> 00:05:31,520
I mean, even on Stack Overflow, out of curiosity, I checked if it's possible to batch SSH to machines to grab and tail.

56
00:05:31,780 --> 00:05:32,500
Oh, right, right, right.

57
00:05:32,760 --> 00:05:34,120
But, no, I mean, don't do that.

58
00:05:34,200 --> 00:05:34,840
It's pointless.

59
00:05:35,020 --> 00:05:36,780
So, we needed a solution.

60
00:05:37,560 --> 00:05:42,500
Well, you laugh, but when I used to work on systems management stuff, like, when would this have been?

61
00:05:42,580 --> 00:05:43,420
Over 10 years ago?

62
00:05:43,520 --> 00:05:45,060
Like, that was one thing we did.

63
00:05:45,380 --> 00:05:49,720
Like, if you wanted to, like, suck out logs, you could always SSH and tail from it.

64
00:05:49,960 --> 00:05:52,240
I mean, obviously, you can do that, and it will work.

65
00:05:52,920 --> 00:05:54,100
This is back in the early 2000s.

66
00:05:54,100 --> 00:05:55,960
But it's cumbersome.

67
00:05:56,080 --> 00:05:56,840
It's difficult, right?

68
00:05:56,900 --> 00:05:59,240
It takes a lot of energy and time.

69
00:05:59,420 --> 00:05:59,600
Yeah.

70
00:05:59,940 --> 00:06:02,620
And it's much better to shift it the other way around.

71
00:06:02,720 --> 00:06:07,260
So, what we did is that, because we had some, let's see, homemade solutions.

72
00:06:07,560 --> 00:06:11,980
We had some cloud somewhere without any abstraction, like Cloud Foundry.

73
00:06:13,740 --> 00:06:18,840
So, we had to invest a lot of time to unify how all of the applications were logging.

74
00:06:19,000 --> 00:06:26,160
Because, of course, as it is in companies, it turns out at some point that every application does the same thing in a different way.

75
00:06:26,540 --> 00:06:29,660
So, if you want to just standardize stuff, you have to invest a lot of time.

76
00:06:30,600 --> 00:06:36,800
So, we had our applications put their logs in, like, always the same place.

77
00:06:37,400 --> 00:06:41,460
And we had a process with log slash forwarder.

78
00:06:41,600 --> 00:06:42,840
Right now, it's called FileBeat.

79
00:06:43,620 --> 00:06:51,760
So, this application was harvesting the logs and putting them all in one place, which was log stash.

80
00:06:52,020 --> 00:06:52,200
Right.

81
00:06:52,580 --> 00:06:55,360
Log stash parsed it into pieces and put it into the database.

82
00:06:55,920 --> 00:07:00,120
So, it's, like, exactly the same thing you can do right now with Cloud Foundry.

83
00:07:00,500 --> 00:07:08,440
Because, by default, all the logs are put, like, into such a place that the Logregator application can take it.

84
00:07:08,860 --> 00:07:11,520
Like, with Logregator, you have all the logs in one place.

85
00:07:12,280 --> 00:07:20,200
So, a lot of the job that we have done, like, manually to standardize, and we spend a lot of bucks on this.

86
00:07:20,800 --> 00:07:22,240
Because, like, it takes time.

87
00:07:22,420 --> 00:07:23,200
And time costs money.

88
00:07:24,140 --> 00:07:26,580
So, you in Cloud Foundry get out of the box for free.

89
00:07:27,240 --> 00:07:28,580
And then you can do whatever you want.

90
00:07:28,640 --> 00:07:31,820
For example, like, you know, cut it into pieces and present this.

91
00:07:32,140 --> 00:07:34,740
You can transform the logs or forward them on to something else.

92
00:07:34,740 --> 00:07:35,020
Whatever you want.

93
00:07:35,540 --> 00:07:38,100
Or just load them up in your plain text editor when you're lonely.

94
00:07:38,360 --> 00:07:38,880
Of course.

95
00:07:39,660 --> 00:07:42,740
It's what we always do, like, when we're lonely.

96
00:07:42,900 --> 00:07:43,540
We re-logs.

97
00:07:44,580 --> 00:07:45,200
So, yeah.

98
00:07:45,380 --> 00:07:46,660
So, that was the solution.

99
00:07:46,820 --> 00:07:52,280
And right now with Sluf, what you get is default logging pattern.

100
00:07:52,380 --> 00:08:01,900
Oh, by the way, I'm pretty sure often people, like, I mean, they're logging, that they're doing their logging, but often their pattern isn't proper.

101
00:08:02,320 --> 00:08:02,980
What does it mean?

102
00:08:04,900 --> 00:08:08,480
Often people forget, for example, to pass the thread name in the logs.

103
00:08:09,100 --> 00:08:09,420
Oh, right.

104
00:08:09,440 --> 00:08:14,340
So, when you're doing, let's say, some parallel processing or stuff, you're totally blind.

105
00:08:14,660 --> 00:08:17,160
Because you don't even know which thread did that.

106
00:08:17,280 --> 00:08:18,940
Like, you have a log, then you have another one.

107
00:08:20,020 --> 00:08:21,420
And you don't know, was it sequential?

108
00:08:21,720 --> 00:08:22,500
Was it in parallel?

109
00:08:22,760 --> 00:08:24,140
You have no idea because you don't log the thread.

110
00:08:24,140 --> 00:08:25,140
Because you don't have the thread in there.

111
00:08:25,320 --> 00:08:25,620
Yeah, yeah.

112
00:08:25,620 --> 00:08:28,780
So, in Sluf, we provide a default logging pattern.

113
00:08:29,280 --> 00:08:30,660
So, we don't have to think about this.

114
00:08:31,360 --> 00:08:39,180
And, basically, if you have Sluf in every single application that you have, you have standardized the logging pattern because you have it out of the box.

115
00:08:39,280 --> 00:08:44,700
Then, if you're using Loggregator in Cloud Foundry, you standardize the way logs are aggregated.

116
00:08:45,460 --> 00:08:54,420
And, you know, it takes a lot of work to actually properly instrument the code to pass the tracing information, like, between threads, context, libraries.

117
00:08:55,160 --> 00:08:59,060
It was hectic work and still is because libraries change.

118
00:08:59,220 --> 00:09:02,720
There are some special use cases, so we have to invest a lot of time.

119
00:09:03,440 --> 00:09:07,760
So, it's better to use Sluf than manually try to do all of that yourself.

120
00:09:07,960 --> 00:09:07,980
Right.

121
00:09:08,160 --> 00:09:19,060
And, I mean, you're getting to what it seems like the, at least from my perspective of having worked on this stuff a long time ago, the core hard part is, there's two things at least.

122
00:09:19,100 --> 00:09:19,780
I'm sure there's more.

123
00:09:20,240 --> 00:09:24,980
But, one, there's instrumenting the code so it puts out the right ID, so it can get the ID.

124
00:09:25,520 --> 00:09:26,460
And propagate it.

125
00:09:26,580 --> 00:09:26,740
Yeah.

126
00:09:26,840 --> 00:09:30,440
And then, also, not only so it can get it, but so that it can use it.

127
00:09:30,620 --> 00:09:34,680
And so, like, that's, like, anytime you have to have developers do something, it's a world of hurt.

128
00:09:35,620 --> 00:09:35,820
Yes.

129
00:09:35,960 --> 00:09:37,880
I mean, that was the crucial thing.

130
00:09:38,020 --> 00:09:43,880
But getting a framework in place that works across many different services and even types of code.

131
00:09:44,100 --> 00:09:50,560
And that kind of gets to the second thing is there's a standard of how you think about and actually do this, right?

132
00:09:50,580 --> 00:09:52,700
Whether it's the log format or other things like that.

133
00:09:52,760 --> 00:10:00,200
But those two things together, then you can finally, like, that's always the problem, at least I would encounter with monitoring, is just instrumenting the stuff.

134
00:10:00,600 --> 00:10:00,680
Right?

135
00:10:00,740 --> 00:10:07,640
And then, as a systems management vendor, all the value that you bring is when the code isn't instrumented.

136
00:10:07,820 --> 00:10:09,580
And you make up for the lack of that.

137
00:10:10,260 --> 00:10:13,000
But, anyways, instrumenting the code is incredibly hard.

138
00:10:13,300 --> 00:10:13,980
Yes, it is.

139
00:10:14,160 --> 00:10:20,280
And, obviously, we haven't done everything in the perfect way.

140
00:10:20,340 --> 00:10:21,820
For sure, there's going to be some issues.

141
00:10:22,200 --> 00:10:23,440
But we are quite stable.

142
00:10:23,880 --> 00:10:26,180
And there are not that many bugs, fortunately.

143
00:10:26,500 --> 00:10:30,860
And we try to really be immediate with fixing them.

144
00:10:31,540 --> 00:10:33,560
But, in general, it's pretty stable.

145
00:10:33,700 --> 00:10:37,120
So, I'm really happy with the quality of instrumentation that we have at the moment.

146
00:10:38,980 --> 00:10:40,720
Yeah, and we support quite a few things.

147
00:10:40,880 --> 00:10:50,680
And we give, let's say, easy, we provide an easy way how to, let's say, instrument new stuff that we don't instrument at the moment.

148
00:10:50,820 --> 00:10:56,460
So, there's at least two more things we should talk about before we wrap up this supposedly quick recording here.

149
00:10:56,940 --> 00:11:02,740
One of them is, so, we don't have a terminal here for you to demonstrate it.

150
00:11:02,760 --> 00:11:04,780
But, like, how does it work?

151
00:11:04,780 --> 00:11:11,180
Like, how, like, and I assume there's, like, some service registry thing that is a piece of code.

152
00:11:11,280 --> 00:11:14,660
You're, like, hey, give me the ID of the thing I'm participating in.

153
00:11:14,960 --> 00:11:20,200
And then you're, kind of, like, and it would be a good idea if I logged with this idea and stuff.

154
00:11:20,420 --> 00:11:29,340
And then, second, like, how, like, between not only different processes, but different types of languages and services and stuff, how does that work?

155
00:11:29,560 --> 00:11:30,260
Okay, so.

156
00:11:30,900 --> 00:11:31,740
Like, so how does it work?

157
00:11:31,920 --> 00:11:32,460
How does it work?

158
00:11:32,460 --> 00:11:40,600
So, in fact, if you're not creating anything custom, like, for example, you don't create a custom span.

159
00:11:40,920 --> 00:11:44,100
So, like, a local span, in fact.

160
00:11:44,200 --> 00:11:46,680
So, a span is an operation.

161
00:11:46,820 --> 00:11:48,820
For example, an RPC call is a span.

162
00:11:49,080 --> 00:11:52,520
So, it's a bounded operation that has its start and finish.

163
00:11:52,620 --> 00:11:53,380
It has a duration.

164
00:11:54,960 --> 00:11:59,040
So, if you don't create a manual one, you don't have to do anything.

165
00:11:59,600 --> 00:11:59,820
Right.

166
00:12:00,040 --> 00:12:01,180
It just happens out of the box.

167
00:12:01,260 --> 00:12:03,660
It's passed on between threads.

168
00:12:04,840 --> 00:12:07,520
It's added to headers in the requests.

169
00:12:08,480 --> 00:12:19,260
And then, basically, because we can wrap, or when you're running in an environment like this, you can wrap the HTTP calls, and then you could kind of wrap all the calls down into the system if there's some kind of runtime that supports it.

170
00:12:19,620 --> 00:12:22,040
And do the old thing of, like, begin and end, begin and end.

171
00:12:22,180 --> 00:12:22,580
Precisely.

172
00:12:22,680 --> 00:12:30,440
So, since we are Zipkin compatible, you can have, for example, the Spring Cloud projects, like your Spring Cloud app.

173
00:12:32,200 --> 00:12:40,100
And you can send a request to, let's say, a Go app, which is Zipkin compatible, and it's going to work.

174
00:12:40,440 --> 00:12:42,260
Because the Go app is going to take the request.

175
00:12:42,840 --> 00:12:46,920
The headers are Zipkin compatible, so it knows what it's supposed to do.

176
00:12:47,420 --> 00:12:54,060
And let's say the Go app is sending the request to the Java app again, the Spring Boot app again.

177
00:12:54,160 --> 00:12:55,240
So, we take it over.

178
00:12:55,580 --> 00:12:58,020
And since we're Zipkin compatible, everything works fine.

179
00:12:58,120 --> 00:13:01,360
So, this is, let's say, the answer to your second question.

180
00:13:01,360 --> 00:13:05,220
And does that work across, like, event queues and things like that, or is it only HTTP?

181
00:13:05,220 --> 00:13:12,740
We do support Spring integration, so we do pass the headers and messages.

182
00:13:13,120 --> 00:13:13,720
Oh, okay, okay.

183
00:13:13,940 --> 00:13:20,860
So, we do support, for example, if you're using RabbitMQ or something like that, the headers are going to be appended to messages.

184
00:13:21,380 --> 00:13:26,100
And then, if you're Zipkin compatible on the other side, you can take the headers from the message on the other side and go.

185
00:13:26,100 --> 00:13:37,080
So, I guess, to try to painfully generalize it, whatever kind of, I don't know, process context you're operating in, you have a span in.

186
00:13:37,120 --> 00:13:46,040
Whether it's over HTTP or a message bus or whatever else we might think of, there's sort of like a bag of metadata associated with it.

187
00:13:46,040 --> 00:13:51,940
And part of what Sleuth and Zipkin does is, like, inserts a thing into that bag.

188
00:13:52,120 --> 00:13:53,100
And extracts it.

189
00:13:53,140 --> 00:14:04,880
Yeah, and then any process in there, I assume there's another sort of, like, standardized, if not sort of libraries that help you with this, is like, anytime you have a piece of code, you can, like, look in that bag and figure out what your ID is.

190
00:14:05,180 --> 00:14:10,260
Precisely. So, in terms of Sleuth, you have a span object on which you can get Trace ID.

191
00:14:10,860 --> 00:14:11,100
Right.

192
00:14:11,340 --> 00:14:15,180
So, this is this correlation ID on this whole business operation.

193
00:14:16,120 --> 00:14:25,240
So, that way you get it, like, immediately inside the code if you need to, for example, log, I don't know, do something about this Trace ID.

194
00:14:25,240 --> 00:14:29,740
Because, in the logs, the Trace ID is always passed in if you're using Sleuth.

195
00:14:30,340 --> 00:14:32,700
So, always you get this value in the logs.

196
00:14:32,920 --> 00:14:45,200
So, afterwards, for example, in Kibana, you can search by Trace ID and you get all the logs from every single service that took part in this one big, let's call it, transaction.

197
00:14:45,400 --> 00:14:47,220
So, that accomplishes two things we were talking about earlier.

198
00:14:47,220 --> 00:15:01,600
So, one, if the developers do nothing, but they're running in this, like, a runtime that supports all of this, you at least can pull out from those log files, like, here's all the logging that happened, that are related to this, and organize it by a span, essentially.

199
00:15:01,960 --> 00:15:02,960
By a Trace.

200
00:15:03,120 --> 00:15:03,420
Yeah, yeah.

201
00:15:03,540 --> 00:15:03,940
By a Trace.

202
00:15:04,000 --> 00:15:06,200
Yeah, you can order by the same Trace.

203
00:15:06,200 --> 00:15:16,920
So, if in one, let's call it, business transaction, there are 100 hops between 100 services, then you're going to see all of those 100 services and their logs.

204
00:15:17,220 --> 00:15:17,480
Right.

205
00:15:17,580 --> 00:15:24,900
If you provide, like, I'm searching via this Trace ID, because this is what correlates all the services, the same value of a Trace ID.

206
00:15:25,040 --> 00:15:25,400
That's right.

207
00:15:25,740 --> 00:15:29,560
So, one Trace ID, but a lot of spans underneath.

208
00:15:29,940 --> 00:15:30,280
Got it.

209
00:15:30,280 --> 00:15:30,840
Yeah, yeah, yeah.

210
00:15:30,840 --> 00:15:39,400
Yeah, I've seen people do similar things with, like, Splunk Magic and other search, where, you know, you find some way of uniquely identifying all this stuff and filter down to it, essentially.

211
00:15:39,440 --> 00:15:40,020
So, this is exactly.

212
00:15:40,320 --> 00:15:46,780
Actually, at SpringOne, I'm going to show it in a demo together with, actually, Kibana.

213
00:15:47,300 --> 00:15:48,740
So, we're going to have a couple of apps.

214
00:15:48,940 --> 00:15:54,940
We're going to log some stuff in the apps and then present them in Kibana and sort them via, filter them via Trace ID.

215
00:15:55,220 --> 00:15:55,900
Ah, all right.

216
00:15:55,940 --> 00:15:57,800
So, the last question before we wrap up.

217
00:15:57,920 --> 00:16:00,000
So, you got all this technology, right?

218
00:16:00,000 --> 00:16:02,180
And we started with, like, what drove you to use it and everything.

219
00:16:02,180 --> 00:16:08,500
So, how would you describe, like, the theory of monitoring that this enables for you?

220
00:16:08,680 --> 00:16:22,080
Like, what – like, if I'm someone responsible for monitoring, like, one of these wacky microservices-driven systems that maybe also has non-microservices stuff, but it's all Zipkin-enabled, like, what do I – how do I use it?

221
00:16:22,120 --> 00:16:27,640
What does it feel like when I'm using sleuth-driven stuff to do my monitoring and remediation jobs?

222
00:16:27,640 --> 00:16:33,100
So, when we're talking about Zipkin, Zipkin is a tool to monitor – to find latency problems.

223
00:16:33,820 --> 00:16:37,000
So, I would separate two things.

224
00:16:37,000 --> 00:16:43,000
Let's say, measuring, for example, KPIs from measuring latency.

225
00:16:43,000 --> 00:16:56,340
So, you can, for example, if you have a critical part in your system that's really important for you from the business perspective, you can wrap it in a custom span, a local span, and you're going to see the exact duration of this part.

226
00:16:56,340 --> 00:17:03,520
You can add tags, so key value pairs that will, like, help you understand more of what's going on in your system.

227
00:17:03,520 --> 00:17:19,140
But the thing is that with Zipkin, you have also sampling, because what that means is that not 100% of, let's say, traces are going to be sent to Zipkin, because you would run out of disk space.

228
00:17:19,520 --> 00:17:24,700
So, by default, by default, we sample 10%.

229
00:17:24,700 --> 00:17:30,700
You can switch on to 100, so every single trace, every single span will be sent to Zipkin.

230
00:17:30,700 --> 00:17:38,180
And in, like, how many, I don't know what to call them at this point, but how many user interactions, like, would you make that cutoff at?

231
00:17:38,240 --> 00:17:43,180
Like, let's say, in an application, and, like, people, there's, like, 100 uses of it in a day.

232
00:17:43,580 --> 00:17:50,580
I assume I would probably collect 100% of stuff, versus, like, there's, like, 100,000 uses of it.

233
00:17:50,580 --> 00:17:56,020
Of course, if you have, like, a bazillion requests per second, I would sample it, obviously, right?

234
00:17:56,020 --> 00:17:58,500
Because, yeah, I mean, you can have problems.

235
00:17:58,640 --> 00:18:04,340
I mean, if you think you won't, and you think that it makes sense, like, sample 100%.

236
00:18:04,340 --> 00:18:13,700
And then you can, like, profit from that in that way that you're going to see every single trace in Zipkin, and you can see that the crucial part of your system takes that much time.

237
00:18:15,040 --> 00:18:33,720
And if that is crucial for you, what I would do is I would, for example, wrap it in a, I don't know, for example, CodeHale metric sort of, let's say, meter or counter or whatever, and then measure that with this tool.

238
00:18:33,820 --> 00:18:45,800
Why? Because then you can, for example, you send it, for example, to Graphite, and then you can have different tools that allow you to put a learning on that.

239
00:18:46,520 --> 00:18:46,640
Right.

240
00:18:46,780 --> 00:18:58,900
So, for example, you have a crucial metric for you, I don't know, amount of requests per second, whatever, let's imagine that, or no, even let's be more pragmatic, amount of money that the current service gives you, for instance.

241
00:18:59,160 --> 00:19:01,920
Let's assume that you can monetize the stuff.

242
00:19:02,820 --> 00:19:18,760
And then you create a metric with which you count the money, and then you can, let's say, put a learning that if you go below certain value, send an email or send a message to Slack or whatever, ping via PagerDuty or something.

243
00:19:18,880 --> 00:19:25,080
So there are tools that can allow you to do so, and that makes more sense because this is typical metric stuff.

244
00:19:25,480 --> 00:19:26,980
What Zipkin is used for is latency.

245
00:19:27,100 --> 00:19:30,360
For example, a user clicks around your system and says, hey, it's slow.

246
00:19:31,000 --> 00:19:31,680
What is wrong?

247
00:19:31,880 --> 00:19:32,460
Right, right, right.

248
00:19:32,580 --> 00:19:39,900
And then you can see, ah, yeah, and you can see the bar graphs, and you can see, for example, immediately that you have a very long bar.

249
00:19:40,400 --> 00:19:42,600
So it means that the request was very slow.

250
00:19:42,820 --> 00:19:43,480
Why is it so?

251
00:19:44,120 --> 00:19:46,220
And then you can click around, figure it out.

252
00:19:46,360 --> 00:19:48,320
So I would discern those two things.

253
00:19:48,320 --> 00:19:49,180
Yeah, that makes sense.

254
00:19:49,600 --> 00:20:00,760
There's sort of like point, not point in time, but single point monitoring of a metric versus what I would unfairly categorize as like weather reports, right?

255
00:20:00,800 --> 00:20:04,160
Like in the sense of like, I want to know how the system is doing in aggregate.

256
00:20:04,380 --> 00:20:05,560
Like your example is good.

257
00:20:05,640 --> 00:20:07,100
It's like people keep telling me it's slow.

258
00:20:07,280 --> 00:20:10,500
Like I look at the logs and I don't know.

259
00:20:10,900 --> 00:20:12,760
Precisely, you don't see anything in the logs.

260
00:20:12,760 --> 00:20:19,860
You would have to log every single method, call or something like that, the count, let's say the duration of a single call.

261
00:20:19,980 --> 00:20:20,440
Right, right, right.

262
00:20:20,440 --> 00:20:23,000
And this is exactly what you get with Zipkin.

263
00:20:23,260 --> 00:20:23,720
Right, right.

264
00:20:23,780 --> 00:20:33,600
So, I mean, those are the kind of like the two types of modes that you would be like monitoring a single point of things essentially and then monitoring kind of the overall health of a system.

265
00:20:33,860 --> 00:20:34,180
Precisely.

266
00:20:34,280 --> 00:20:38,940
It's almost like quantitative and qualitative.

267
00:20:39,520 --> 00:20:43,480
Like the quantitative thing is like here's a bunch of data points that are exactly this.

268
00:20:43,960 --> 00:20:47,060
And qualitative is like, I don't know, it feels okay or it feels bad.

269
00:20:47,900 --> 00:20:48,640
Yes, precisely.

270
00:20:49,640 --> 00:20:53,760
So, that's what I would discern KPIs from the overall state of the system.

271
00:20:53,940 --> 00:20:54,180
Great.

272
00:20:54,320 --> 00:20:54,640
Well, great.

273
00:20:54,740 --> 00:20:56,560
Well, that's fantastic.

274
00:20:56,860 --> 00:20:57,580
That's a good quick overview.

275
00:20:57,740 --> 00:21:02,220
So, tell people like, you know, where you are on Twitter or you've got a lot on your blog too.

276
00:21:02,340 --> 00:21:06,000
I often interview people and they're like, I haven't updated my blog in five years.

277
00:21:06,260 --> 00:21:07,860
But I think yours is a lot more recent.

278
00:21:07,920 --> 00:21:09,840
I'm trying to be like a frequent blogger.

279
00:21:10,340 --> 00:21:14,020
So, like I said, my name is Marcin Grzeisztak.

280
00:21:14,120 --> 00:21:16,100
So, my Twitter handle is at mgrzeisztak.

281
00:21:16,100 --> 00:21:19,380
If you're not Polish, you're not going to even like find it.

282
00:21:19,640 --> 00:21:23,280
So, I suggest you go to my blog, toomuchcoding.com.

283
00:21:23,620 --> 00:21:24,380
That's right.

284
00:21:24,800 --> 00:21:27,340
Like, you have very chances of remembering that.

285
00:21:27,800 --> 00:21:29,240
And over there you can find my Twitter handle.

286
00:21:30,460 --> 00:21:35,100
And, yeah, I try to blog frequently about, let's say, microservices as such.

287
00:21:35,340 --> 00:21:39,100
Right now I'm trying to blog also on the Spring IO blog.

288
00:21:40,280 --> 00:21:45,580
And I'm trying to blog about stuff like answering difficult questions that arise.

289
00:21:45,940 --> 00:21:49,940
Like, how to do zero downtime deployment, for example, in terms of the database.

290
00:21:50,500 --> 00:21:53,300
Because with Cloud Foundry you can do magical stuff like CF Push and that's it.

291
00:21:53,340 --> 00:21:53,720
It's working.

292
00:21:54,000 --> 00:21:56,320
But you have to remember about the database, for example.

293
00:21:56,440 --> 00:21:57,000
Yeah, always the data.

294
00:21:57,340 --> 00:21:59,980
Yeah, how to deploy microservices, for example.

295
00:21:59,980 --> 00:22:00,100
Right, right.

296
00:22:00,220 --> 00:22:01,960
How the pipeline should look like.

297
00:22:02,140 --> 00:22:03,860
So, you can check out my blog.

298
00:22:05,100 --> 00:22:05,720
Well, great.

299
00:22:05,820 --> 00:22:07,460
You could find a lot of answers there, I guess.

300
00:22:07,600 --> 00:22:08,620
Yeah, well, I appreciate it.

301
00:22:08,880 --> 00:22:17,500
We'll, as always, have this at soundcloud.com slash pivotalconversations and at the blog at pivotal.io slash podcast.

302
00:22:17,800 --> 00:22:19,160
Maybe you need to put a blog in front of there.

303
00:22:19,320 --> 00:22:19,620
I don't know.

304
00:22:19,660 --> 00:22:21,220
Just search for pivotalconversations.

305
00:22:21,560 --> 00:22:22,520
People will find it.

306
00:22:22,920 --> 00:22:24,040
We'll see everyone next time.

307
00:22:24,040 --> 00:22:24,040
Thank you.