1
00:00:00,000 --> 00:00:06,240
Well this week Richard can make it so we have uh I don't know perhaps the uh the the second most

2
00:00:06,240 --> 00:00:09,620
second or third most popular person aside from me on this podcast

3
00:00:09,620 --> 00:00:20,840
yeah this is Andrew Andrew Clay Schaefer we're gonna I'm I'm standing in that's right I I think

4
00:00:20,840 --> 00:00:27,840
I think uh it'll be great I answered the call that's right yeah I I just uh it occurred to me

5
00:00:27,840 --> 00:00:32,320
that uh because Richard was going to be unavailable for a couple weeks I should get someone else to

6
00:00:32,320 --> 00:00:38,420
co-host because despite my self-proclaimed popularity it's kind of like salt like too

7
00:00:38,420 --> 00:00:44,320
much and it ruins it so we gotta we gotta balance it out with something uh so what's what's what's

8
00:00:44,320 --> 00:00:53,280
the spice of mine uh sugar is that is that right I am definitely not sugar have you talked to me

9
00:00:53,280 --> 00:01:00,240
maybe maybe you're uh uh I don't know what do they got over there uh some sort of like uh

10
00:01:00,240 --> 00:01:05,560
southwestern uh chili yeah are you some of those are you a mixture of chili and garlic on a string

11
00:01:05,560 --> 00:01:15,400
left out to dry in the desert probably that's moving on well uh there you go with that tone

12
00:01:15,400 --> 00:01:20,940
setting so we have a guest on this week why don't you introduce yourself sure uh my name is Todd Pearson

13
00:01:20,940 --> 00:01:29,760
um I was uh a labs pivot back in New York many years ago uh and departed to start a company that

14
00:01:29,760 --> 00:01:35,800
became uh influx data so we were the creators of the open source time series database influx db

15
00:01:35,800 --> 00:01:44,700
um and I did that for the last five years or so and then returned to pivotal um primarily on the

16
00:01:44,700 --> 00:01:50,680
engineering side but my uh sort of subtitle I guess if you could call it is uh subject matter expert for

17
00:01:50,680 --> 00:01:58,780
metrics um effect effectively that just means I'm kind of working cross team to help um sort of bring

18
00:01:58,780 --> 00:02:03,720
metrics to the forefront and help with best practices and kind of make sure that we have

19
00:02:03,720 --> 00:02:11,100
some sort of an overarching theme that we're adhering to as far as metrics go across all of cloud

20
00:02:11,100 --> 00:02:18,120
found tree so that was before sorry just started in before we before we get into metrics I had a few

21
00:02:18,120 --> 00:02:25,100
at a few uh what's the opposite of a follow-on question a lead-on question one uh so uh we got

22
00:02:25,100 --> 00:02:32,260
leading the witness there you go so so we had a uh uh this is kind of like a follow-up to the uh

23
00:02:32,260 --> 00:02:37,880
nice Andrew's on to the google sre book uh book review uh that we did because there is there's quite a bit

24
00:02:37,880 --> 00:02:43,480
about uh not only like uh monitoring and metrics but also time series data or data however you want

25
00:02:43,480 --> 00:02:48,060
to say it I always suffer from growing up watching star treks the next generation so I always say it

26
00:02:48,060 --> 00:02:53,860
wrong but uh anyhow which leads me to my second thing now this is a little bit of like uh you know

27
00:02:53,860 --> 00:02:59,920
yokel Coté asking this but since you worked on some time series data stuff and I think Andrew and I

28
00:02:59,920 --> 00:03:05,900
discussed this back back a while I'm curious about your take of like so what's what's the deal with

29
00:03:05,900 --> 00:03:11,800
like all the hype around time series data stuff like is it actually like something brand new that

30
00:03:11,800 --> 00:03:16,360
hasn't existed before or is it just like oh this is a really cool thing that we should all be using

31
00:03:16,360 --> 00:03:23,540
more and as context so back when I did real work and I programmed like I worked at BMC on a BMC

32
00:03:23,540 --> 00:03:29,680
performance manager a gloriously named product and you know we did monitoring but uh back then people

33
00:03:29,680 --> 00:03:34,660
wouldn't really use the phrase time series data but it seems to have been like it's like a big deal

34
00:03:34,660 --> 00:03:39,740
nowadays so maybe uh I just you know went off and learned how to do powerpoint and stop paying

35
00:03:39,740 --> 00:03:43,760
attention and people have always talked about that but it seems like it's a phrase that happens a lot

36
00:03:43,760 --> 00:03:51,620
more than uh than I'm used to yeah it's definitely been something that has kind of entered the I don't

37
00:03:51,620 --> 00:03:56,780
know popular vernacular more in the in the last few years and it was something that we struggled with a

38
00:03:56,780 --> 00:04:03,080
bit at at influx data because um not everyone really understands what it is yet um you tell them

39
00:04:03,080 --> 00:04:09,340
metrics and analytics and they kind of get a sense for it but time series data as a larger category still

40
00:04:09,340 --> 00:04:16,900
seems a little bit vague and uh generic to a lot of people but really it's it's not all that new it's

41
00:04:16,900 --> 00:04:21,780
it's primarily been in the financial arenas it's if you talk to economists and things that they're much

42
00:04:21,780 --> 00:04:28,820
more familiar with time series data because they've been using it for um you know forecasting and analysis

43
00:04:28,820 --> 00:04:36,620
for you know decades at least um and a lot of uh the earlier work in in time series databases was for

44
00:04:36,620 --> 00:04:41,420
financial analysis so if you're thinking about um stock trades and things like that where you have a

45
00:04:41,420 --> 00:04:46,280
high volume of transactions that have to be stored and processed those are those were some of the first

46
00:04:46,280 --> 00:04:51,320
time series databases that really were were popularized in software um but I would say that

47
00:04:51,320 --> 00:04:56,660
there's definitely been a a growth over the last 10 years or so in kind of modern software development

48
00:04:56,660 --> 00:05:01,480
and how how people think about time series data and one of the I think one of the things that really

49
00:05:01,480 --> 00:05:09,240
pushed pushed it into our I guess point of view and um I guess larger like software development

50
00:05:09,240 --> 00:05:13,700
in in general was uh you know new relic became something that people could just sort of drop into

51
00:05:13,700 --> 00:05:19,520
the the ruby on rails application with a single a single library and you had a lot of people who

52
00:05:19,520 --> 00:05:24,820
were relatively novice web developers that were kind of like growing up with rails that suddenly had

53
00:05:24,820 --> 00:05:31,100
these powerful tools that they could get um really easily and it that sort of became a much more common

54
00:05:31,100 --> 00:05:37,660
with the rise of you know sass monitoring tools but um lucerne the guy that started new relic had a

55
00:05:37,660 --> 00:05:44,640
company before new relic called wiley technologies he sold to um i think computer associates you know

56
00:05:44,640 --> 00:05:49,480
several years before starting new relic and that was also the same type of you know performance

57
00:05:49,480 --> 00:05:55,520
monitoring metrics collection but it was all for the enterprise java stack so it's been around it's

58
00:05:55,520 --> 00:06:00,640
just that now the availability of tools like new relic and the popularization of things like graphite

59
00:06:00,640 --> 00:06:05,000
which was you know written in python and uh used heavily at companies like etsy

60
00:06:05,000 --> 00:06:10,580
having those things kind of floating around and establish more of these best practices has really

61
00:06:10,580 --> 00:06:14,400
made them something that i think people are feeling much more comfortable with and starting to expect

62
00:06:14,400 --> 00:06:20,220
rather than uh kind of just like an experimental thing that people are using to to test out their

63
00:06:20,220 --> 00:06:24,340
applications i know that that explanation that that actually makes a lot of sense and kind of

64
00:06:24,340 --> 00:06:30,580
comports with my foggy memories of uh i mean tell me if this summary is wrong is uh uh as you're saying

65
00:06:30,580 --> 00:06:36,580
we just have much more uh powerful tools and just raw computation and storage that we can apply to it

66
00:06:36,580 --> 00:06:41,120
so you can do a lot more with with time-based series than uh back when i was worrying about

67
00:06:41,120 --> 00:06:46,080
mibs and stuff like that which uh certainly maps to what seems to be available out there

68
00:06:46,080 --> 00:06:52,200
anyone doing monitoring was always storing time series data i think that the thing that's changed is

69
00:06:52,200 --> 00:06:58,580
people we have the dark ages of you know the sql databases which you can also use as queues by the

70
00:06:58,580 --> 00:07:03,720
way but when you think about the time series databases and some of the stuff that's out there with

71
00:07:03,720 --> 00:07:11,680
influx or imgb or even cassandra for that matter like you you can think about the queries and the and

72
00:07:11,680 --> 00:07:16,980
the questions you want to ask and they're optimized for for asking and answering those kind of queries

73
00:07:16,980 --> 00:07:22,660
about you know the trends in those in those functions that are stored as time series instead of

74
00:07:22,660 --> 00:07:28,320
whatever relational algebra right yeah that meant yeah and one thing i was going to point out is

75
00:07:28,320 --> 00:07:33,380
that um you brought up relational databases kind of in the in the scope of time series databases and

76
00:07:33,380 --> 00:07:39,600
most of new relic for the first several years at least was powered entirely by mysql it was just

77
00:07:39,600 --> 00:07:45,880
like a massively sharded mysql setup that that's how that's how everyone starts yeah that's how it

78
00:07:45,880 --> 00:07:50,000
starts because it works i mean it's it's well known and i think a lot of the custom time series

79
00:07:50,000 --> 00:07:55,560
databases have been a much more like fancy modern thing and there's there's a need for it but the

80
00:07:55,560 --> 00:08:02,840
need comes much later when you start to scale i guess i guess mysql is the uh the trash 80 of this

81
00:08:02,840 --> 00:08:08,520
generation of systems management people it's it's like what everyone grew up on yeah i always like

82
00:08:08,520 --> 00:08:14,260
stories of the legendary wiley that was always a uh a fun company back then excellently named as well

83
00:08:14,260 --> 00:08:21,420
uh also all sorts of meanings well then that brings us to uh to to the nominal topic at hand so uh like

84
00:08:21,420 --> 00:08:26,820
you're saying you you started off uh as as a you know working in pivot labs for a while went off to

85
00:08:26,820 --> 00:08:32,060
uh explore what we were just talking about and then you came back to be one of my favorite acronyms

86
00:08:32,060 --> 00:08:36,320
a smee i think i think that might have been one of the rejected names for smurfs when they were

87
00:08:36,320 --> 00:08:41,800
thinking that up but to a subject matter expert all around and so so first of all describe what

88
00:08:41,800 --> 00:08:49,100
metrics is like kind of what uh what it does and what uh what need it fills in you know that sort of

89
00:08:49,100 --> 00:08:55,380
context for it sure so um specifically are you asking about pcf metrics that sort of product

90
00:08:55,380 --> 00:09:01,680
absolutely uppercase metrics uppercase metrics another way to frame it is now we have a time series

91
00:09:01,680 --> 00:09:08,460
database what should we do with it well surprisingly or not so surprisingly our time series database is

92
00:09:08,460 --> 00:09:15,760
still mysql under the under the hood for now uh because it it does what we need so um what what

93
00:09:15,760 --> 00:09:21,340
pcf metrics is trying to do right now um and i'll i'll explain like kind of where we are and where

94
00:09:21,340 --> 00:09:27,200
metrics as like a broader concept is headed inside of cloud foundry but the pcf metrics team currently

95
00:09:27,200 --> 00:09:33,500
is focusing on uh was initially focused mostly on instance level metrics kind of pulling cpu memory

96
00:09:33,500 --> 00:09:40,760
disc stats from um cloud foundry instances and the new release that's coming out uh probably

97
00:09:40,760 --> 00:09:48,040
somewhere around the same time as 1.11 uh is going to allow application developers to easily emit

98
00:09:48,040 --> 00:09:55,660
metrics from within their code so um there's some some hooks that we've got uh in spring boot uh through

99
00:09:55,660 --> 00:10:01,040
using drop wizard metrics as well um that allow application developers to easily wrap endpoints

100
00:10:01,040 --> 00:10:08,360
um just wrap blocks of code with timers and things like that and those will all automatically get uh sent

101
00:10:08,360 --> 00:10:15,200
over to a metrics api that we were publishing and then that will get pulled into the pcf metrics application

102
00:10:15,200 --> 00:10:21,360
and from there you can build custom dashboards and set up charts to look at metrics the way that you

103
00:10:21,360 --> 00:10:26,520
want similar to what you do with datadog or something like that but it's all sort of automatically configured

104
00:10:26,520 --> 00:10:32,840
within the cloud foundry infrastructure so so there there's a little aspect just to uh just to call it

105
00:10:32,840 --> 00:10:36,920
out that that uh i don't know it's subtle but important and andrew and i were talking about this

106
00:10:36,920 --> 00:10:42,900
when we were talking about the sre book is the uh the notion that a developer could actually instrument

107
00:10:42,900 --> 00:10:49,360
their application is is sort of phenomenal it's almost an advance in uh in in the multi-decade

108
00:10:49,360 --> 00:10:54,940
journey of systems management and i and i think uh especially if you're kind of angling towards those

109
00:10:54,940 --> 00:10:59,540
weekly or daily releases that that we try to get people towards with the whole uh pivotal approach

110
00:10:59,540 --> 00:11:05,020
to things that uh i don't know i mean that just opens up so many things that that are nice like

111
00:11:05,020 --> 00:11:11,360
previously you would have to treat um uh you know the the resource that you are monitoring and managing

112
00:11:11,360 --> 00:11:17,280
as a black box but now with like the built-in notion that you can just add in i guess not whatever

113
00:11:17,280 --> 00:11:23,580
you want but more or less whatever you want seems like a huge benefit and then and then also it drives

114
00:11:23,580 --> 00:11:28,960
kind of a change to how you think about i i would imagine coding up something like uh all the metric

115
00:11:28,960 --> 00:11:34,620
stuff to make it more developer friendly yeah and i think it's it's something that um i think has been a

116
00:11:34,620 --> 00:11:39,040
little bit more common like if you look at datadog they've been they've been on a bit of a tear for

117
00:11:39,040 --> 00:11:45,120
the last few years because like you're saying that concept of being able to allow developers to sort

118
00:11:45,120 --> 00:11:50,640
of fire metrics off into a service and have them get collected and then make it easy to surface those

119
00:11:50,640 --> 00:11:55,740
is is really appealing and it does it does make it a lot easier to kind of you know you you may have

120
00:11:55,740 --> 00:12:02,560
code that's functional but you may not be able to without load see how it's going to behave um once real

121
00:12:02,560 --> 00:12:08,600
users start hitting it so i think being able to have those things in place um is is really powerful

122
00:12:08,600 --> 00:12:13,080
as a as a developer and as a team when you want to start troubleshooting and the nice thing about

123
00:12:13,080 --> 00:12:19,740
having it you know easily deployable inside of cloud foundry with basically no configuration is that

124
00:12:19,740 --> 00:12:25,820
you don't have to sit there and watch your datadog bill as users just start sending more and more

125
00:12:25,820 --> 00:12:31,640
stuff because that it has a tendency to add up rather quickly um and this is a nice way for people

126
00:12:31,640 --> 00:12:36,260
to kind of bridge that gap between having something that's can be prohibitively expensive at a certain

127
00:12:36,260 --> 00:12:41,820
point um versus having something that you can basically get out of the box for for free

128
00:12:41,820 --> 00:12:48,740
well it's it's just part of the value of the platform but this this observability this kind

129
00:12:48,740 --> 00:12:54,600
of telemetry is critical no one's going to have a better idea or understanding of the app's health

130
00:12:54,600 --> 00:12:59,400
and the developers who wrote it in the first place so it's the best it's the best time to add that

131
00:12:59,400 --> 00:13:04,620
yeah and i think if you can get um you get your teams thinking that way and thinking about

132
00:13:04,620 --> 00:13:12,740
the instrumentation as almost almost part of a documentation slash testing kind of mindset i think

133
00:13:12,740 --> 00:13:17,840
it can go a long way in helping the the reliability of apps especially when you're deploying them

134
00:13:17,840 --> 00:13:23,040
inside of a cloud infrastructure and you may not necessarily be able to as easily you know watch

135
00:13:23,040 --> 00:13:26,740
them and and monitor them because they're inside of containers or whatever they're not quite as

136
00:13:26,740 --> 00:13:32,020
accessible from the outside one one magic way to give them that mindset is to make them wear

137
00:13:32,020 --> 00:13:38,580
pagers for their app that's true then they'll monitor nothing on on the topic of you know on in the

138
00:13:38,580 --> 00:13:44,260
vein of being a subject matter expert so first um someone who would be you know just taking care of

139
00:13:44,260 --> 00:13:49,380
the monitoring and using something like uppercase metrics like in your mind first like what are the

140
00:13:49,380 --> 00:13:53,960
kind of like personas or people who get involved in that and then and then as you delve into that like

141
00:13:53,960 --> 00:14:00,620
what are what are like the things they would want to look at and obviously like uh this is dependent

142
00:14:00,620 --> 00:14:05,320
on the exact type of application but in general like what are the kind of things they would want to

143
00:14:05,320 --> 00:14:09,740
instrument and kind of how they would want to interact with and query this data like what's

144
00:14:09,740 --> 00:14:15,680
what's your view of what the user would like to do as they say sure yeah and i i think um on the on

145
00:14:15,680 --> 00:14:21,320
the topic of personas i feel like so there's there's another sort of part to this whole thing which is

146
00:14:21,320 --> 00:14:27,220
the project that we're sort of working on right now that's you know being dubbed pcf metrics is sort

147
00:14:27,220 --> 00:14:31,740
of like half of what's really going on and there's also another piece that is going to be

148
00:14:31,740 --> 00:14:37,700
platform metrics which will be is being built by a separate team it will live as a separate uh product

149
00:14:37,700 --> 00:14:42,080
for now but the idea is that that sort of covers more of the things that tell you about the health

150
00:14:42,080 --> 00:14:46,380
of your cloud foundry installation are your services up and running so there are things like that that

151
00:14:46,380 --> 00:14:53,540
are geared more towards the the operators as a persona whereas the pcf metrics app that i'm working on

152
00:14:53,540 --> 00:15:00,440
with with this team is more geared towards sort of developers and developers handing things off to

153
00:15:00,440 --> 00:15:05,440
whoever might be on on call so it could be somebody who's part of the team like here we sort of go

154
00:15:05,440 --> 00:15:12,360
through a rotation where developers take on call shifts um for for things on pws but if you have a

155
00:15:12,360 --> 00:15:19,100
dedicated ops or sre team that's kind of looking at these metrics for for different services or have some

156
00:15:19,100 --> 00:15:25,100
sort of uh kpis that they want to make sure that they're hitting um that's sort of the the two

157
00:15:25,100 --> 00:15:30,940
halves to the the real i think user persona of this pcf metrics and generally a lot of the things that

158
00:15:30,940 --> 00:15:35,200
you're looking at like you said will be sort of application dependent but there are always the

159
00:15:35,200 --> 00:15:42,500
sort of soft ones like cpu utilization memory utilization those are kind of like it could be high

160
00:15:42,500 --> 00:15:47,560
it could be low depends on if you know how you're using your resources if that's correct or not

161
00:15:47,560 --> 00:15:52,240
like if you're using redis maybe the memory is always kind of high or maybe you're always cpu bound

162
00:15:52,240 --> 00:15:57,220
um but those are those are the things that some people look at as just sort of a kind of a sniff

163
00:15:57,220 --> 00:16:02,480
test for whether or not things look like they're under or over provisioned and then i think the things

164
00:16:02,480 --> 00:16:06,580
that really have the most value are the ones that we're kind of moving toward towards in this next

165
00:16:06,580 --> 00:16:15,180
release where you can say um how many items are in my redis queue or what's the approximate time

166
00:16:15,180 --> 00:16:21,100
taken to process a job and there are things that you can sort of uh keep an eye on that are sort of part

167
00:16:21,100 --> 00:16:27,600
of either like a processing pipeline or sort of part of your app's overall response cycle that can tell

168
00:16:27,600 --> 00:16:32,760
you really quickly like oh the queue depth has been at 100 for the last week and all of a sudden it's

169
00:16:32,760 --> 00:16:38,040
4 000 something's not getting processed correctly or somebody's slowing down it could network could be

170
00:16:38,040 --> 00:16:42,360
a crashed application and those are the things that i think starting to be able to get

171
00:16:42,360 --> 00:16:48,980
sort of customized views into uh are really where you can start to get a lot of value out of out of

172
00:16:48,980 --> 00:16:56,680
metrics um so you sort of you sort of got the uh uh checking in on the behavior of the systems you're

173
00:16:56,680 --> 00:17:02,360
looking at right like so some of the things you're listing were like uh uh what's the flow of of data

174
00:17:02,360 --> 00:17:07,940
or processes through a pipe or whatever and just kind of making sure that uh it kind of looks normal

175
00:17:07,940 --> 00:17:12,780
i guess or or even when it's not looking normal to kind of check in and see what's going on with it

176
00:17:12,780 --> 00:17:17,120
and and kind of looking at that and then and then there's there's the uh the second thing you it's

177
00:17:17,120 --> 00:17:22,400
kind of the other side of that which is like uh trying to automate the detection of a weird state

178
00:17:22,400 --> 00:17:27,820
like like you were saying you know normally we would expect this result to be like 100 but now

179
00:17:27,820 --> 00:17:34,100
it's 10 000 so maybe that's great or maybe that's terrible and and we should probably figure out what's

180
00:17:34,100 --> 00:17:39,000
going on with that right and i think the the the you sort of said something about sort of automatic

181
00:17:39,000 --> 00:17:44,060
detection um and i feel like there's sort of this holy grail that everybody wants which is

182
00:17:44,060 --> 00:17:51,420
sort of uh automatic anomaly detection of error condition in in anything and that's something that's

183
00:17:51,420 --> 00:17:55,360
pretty far off just simply because it's it's so much data to process so there are a few things that

184
00:17:55,360 --> 00:18:00,760
people have done to do sort of statistical correlations between different time series and

185
00:18:00,760 --> 00:18:06,660
some of that's proved useful but um one thing that etsy sort of pioneered or or at least like

186
00:18:06,660 --> 00:18:14,880
logged about extensively was just pure visual inspection you get your dashboards and you you know what

187
00:18:14,880 --> 00:18:18,560
they're supposed to look like and you look at them all the time and as soon as you see something

188
00:18:18,560 --> 00:18:23,960
that looks you know out of whack like and i know especially at etsy they deploy it all the time

189
00:18:23,960 --> 00:18:28,660
and every time they do a deployment engineers would just stare at their dashboards and they'd look for

190
00:18:28,660 --> 00:18:34,060
a spike or a drop or something that looked like it was out of out of what they would expect based on

191
00:18:34,060 --> 00:18:39,760
just a visual observation and i think that that actually could be really powerful um just for that

192
00:18:39,760 --> 00:18:47,200
that quick observation uh that something's going wrong human vision is pretty good at detecting anomalies

193
00:18:47,200 --> 00:18:52,800
so there's there's a bunch of conversations that are going along in the industry and i think

194
00:18:52,800 --> 00:18:58,920
pinnable is in a unique place to help people understand um or or you know progress with respect

195
00:18:58,920 --> 00:19:03,620
to things like the service level objectives i don't know if you've if you've had a chance to read the

196
00:19:03,620 --> 00:19:09,020
their sre book or heard some of the conversations about some of the cre work that we're doing with

197
00:19:09,020 --> 00:19:16,800
google right now but but being able to take and frame an application's behavior in in terms of

198
00:19:16,800 --> 00:19:23,280
service level objectives and then service level indicators is is game changing and it's especially

199
00:19:23,280 --> 00:19:30,840
it's especially lacking in kind of that enterprise segment and in the conversations i have um you know

200
00:19:30,840 --> 00:19:36,100
with a lot with a lot of our customers they're they're they're actually they don't know what their

201
00:19:36,100 --> 00:19:42,280
objectives are they don't know what their indicators are and so being able to help people see that

202
00:19:42,280 --> 00:19:49,240
clearly frame that clearly with a shared metaphor across their organization and then and then also

203
00:19:49,240 --> 00:19:55,540
coming back into our product i think that will that'll help advance and accelerate a lot of the

204
00:19:55,540 --> 00:20:00,800
the benefits that people are getting out of the platform i completely agree with that

205
00:20:00,800 --> 00:20:05,800
and i feel like the i think the google sre book is sort of um i guess in the way that i was saying

206
00:20:05,800 --> 00:20:11,420
that new relic sort of made people start to uh realize that they could come to expect you know

207
00:20:11,420 --> 00:20:16,140
being able to have metrics as sort of a a tool that was always available to them as a developer

208
00:20:16,140 --> 00:20:22,520
i feel like the the sre book has sort of pushed this um this concept of services monitoring into

209
00:20:22,520 --> 00:20:27,360
much more of a mainstream topic i've seen it come up at a lot of different companies now where

210
00:20:27,360 --> 00:20:31,280
you know they're doing a book club they're going through the sre book and people are getting more

211
00:20:31,280 --> 00:20:37,000
familiar with with slos and i think having having the technology and infrastructure to be able to

212
00:20:37,000 --> 00:20:42,760
to instrument your applications made it possible to start having those conversations around what are

213
00:20:42,760 --> 00:20:48,980
the things that we want to look at and how do we how do we define those goals and now it seems like

214
00:20:48,980 --> 00:20:53,960
maybe the the trickiest part in some of these cases is if you're talking about uptime how do you

215
00:20:53,960 --> 00:20:58,500
measure the uptime like now we know how to record it we know how to visualize it but what what thing

216
00:20:58,500 --> 00:21:05,500
in your code do you instrument to like determine yeah with some sort of reliability is the uptime

217
00:21:05,500 --> 00:21:10,680
x like how do i get that out so i think it's giving people a way to start uh experimenting with that

218
00:21:10,680 --> 00:21:16,220
stuff and really uh making it it possible and i think you're right that i think pcf metrics does give

219
00:21:16,220 --> 00:21:21,640
teams you know within cloud foundry an easy way to just have this place to start experimenting with

220
00:21:21,640 --> 00:21:31,000
those kinds of of goals i i think the uptime is is one thing that is maybe nebulous but you can always

221
00:21:31,000 --> 00:21:38,960
figure out what your business does and monitor that right like we we sell this thing if how many did we

222
00:21:38,960 --> 00:21:45,760
sell the last minute how many did we sell or whatever our transactions whatever our meaningful metrics are

223
00:21:45,760 --> 00:21:52,080
and they're going to be contextual contextual to every organization but i think measuring those high

224
00:21:52,080 --> 00:21:59,360
level things often gives you more more insight than than the rest of the system and maybe you want

225
00:21:59,360 --> 00:22:03,480
to drill down when you're troubleshooting but the only thing that matters for your business are did you

226
00:22:03,480 --> 00:22:09,700
sell the thing not not the rest of the numbers so so along those lines i'm curious i mean we'll see

227
00:22:09,700 --> 00:22:14,720
we'll see if this works out but as pulling all that together including what andrew was saying like if

228
00:22:14,720 --> 00:22:21,880
uh i'm remembering uh there's there's one talk that uh who was it uh a comcast person gave at spring

229
00:22:21,880 --> 00:22:28,080
one platform james taylor but not the james taylor uh and uh he was saying you know one of the things

230
00:22:28,080 --> 00:22:33,720
they wanted to fix was preventing people from uh calling into a call center because that basically is

231
00:22:33,720 --> 00:22:38,840
five dollars off the off the top right there and then uh and then the costs just escalate as you're

232
00:22:38,840 --> 00:22:44,940
talking to a human on the phone and so they were putting in place a uh basically uh well i forget

233
00:22:44,940 --> 00:22:49,820
the exact application but it was something to uh help people uh figure out what they would do instead

234
00:22:49,820 --> 00:22:55,460
of calling into a call center i think you know techs and in customers of comcast and so there's an

235
00:22:55,460 --> 00:22:59,700
interesting little metric there of like we would like to monitor how many people are calling into the

236
00:22:59,700 --> 00:23:06,300
call center like because we want that number to be lower and so um how how like if someone were like

237
00:23:06,300 --> 00:23:10,800
coding up their own instrumentation for an application like how would they instrument that

238
00:23:10,800 --> 00:23:16,160
business data if you will into it like how would that surface and kind of get integrated into the

239
00:23:16,160 --> 00:23:21,820
overall health of the service they were doing so i think in the most basic sense um you know some

240
00:23:21,820 --> 00:23:25,680
somewhere in their code they've got this concept of uh you know a call being received

241
00:23:25,680 --> 00:23:31,740
and you know there are a couple like different types of things that you can collect some things are

242
00:23:31,740 --> 00:23:38,380
uh like a metric which is basically like something that has a value at a given point in time so that

243
00:23:38,380 --> 00:23:43,480
could be like cpu utilization memory utilization it's something that's sort of got just a number that

244
00:23:43,480 --> 00:23:48,620
varies over time and then the other side of it is you can also just have things that are more like

245
00:23:48,620 --> 00:23:53,400
events so a call being received is an event and it doesn't really have a value associated with it it's

246
00:23:53,400 --> 00:24:00,060
sort of like a unit of one um and you can just send send those events and then have something on the

247
00:24:00,060 --> 00:24:05,060
back end that's that's receiving them or something that's either like uh this is part of the way that

248
00:24:05,060 --> 00:24:10,260
some of the stuff in spring boot works you've got like a little accumulator basically that will keep

249
00:24:10,260 --> 00:24:15,500
track of how many times you increment a counter and then when you uh flush it once a minute or

250
00:24:15,500 --> 00:24:20,960
whatever it'll send the the count that was received in the past minute and essentially that gets

251
00:24:20,960 --> 00:24:25,520
pushed out to then that becomes like a metric so then it's like how many calls per minute did i get

252
00:24:25,520 --> 00:24:31,980
or whatever and then that sort of gets uh sent out to you know our time series database effectively

253
00:24:31,980 --> 00:24:38,660
and with that then you can suddenly have something that you're able to chart uh over you know over

254
00:24:38,660 --> 00:24:42,940
time you initially start looking at it per minute maybe you're doing an aggregation and you want to

255
00:24:42,940 --> 00:24:47,960
know what is it per hour what is it per week what was it this week versus last week and once you start

256
00:24:47,960 --> 00:24:53,380
collecting that data now you've got this sort of powerful trending that you can do and say like am i

257
00:24:53,380 --> 00:25:00,560
am i decreasing my overall call center volume month over month and you can you can set that stuff up and

258
00:25:00,560 --> 00:25:07,980
right now we're kind of focusing on two week retention periods for pcf metrics but the the goal is be able

259
00:25:07,980 --> 00:25:13,660
to open that up so that you know for some metrics you could store 30 days or 100 days or whatever it is

260
00:25:13,660 --> 00:25:19,360
that you're looking over and then you can make that sort of a business goal looking at month over

261
00:25:19,360 --> 00:25:24,980
month what is what is the the increase or decrease look like so so essentially it's it's the uh same

262
00:25:24,980 --> 00:25:31,360
old integration thing is like well somewhere there needs to be a uh a unit of compute whether it's uh

263
00:25:31,360 --> 00:25:36,100
whether it's a a fun old script or something more sophisticated that's firing off an event or

264
00:25:36,100 --> 00:25:40,920
somehow noting that like a call has come in or whatever and then you're essentially just aggregating that

265
00:25:40,920 --> 00:25:45,960
into some point and then another another little uh part of the system comes along and does something

266
00:25:45,960 --> 00:25:51,000
with that that data if only just displaying it or providing it to somewhere and then and then i would

267
00:25:51,000 --> 00:25:55,120
imagine i mean i guess you could think of it as the uh i mean the classic scenarios i remember from

268
00:25:55,120 --> 00:26:01,760
apm are always uh buying a book and like forget password or whatever yep and uh you could sort of

269
00:26:01,760 --> 00:26:08,760
monitor that your automated forget password system uh was was probably going screwy if all of a sudden you had a

270
00:26:08,760 --> 00:26:14,440
huge spike in people calling in to uh reset their password i mean that's a really simplistic thing

271
00:26:14,440 --> 00:26:19,080
but it it does it does seem like one of those things where you uh you could pull together a lot of

272
00:26:19,080 --> 00:26:24,580
different parts of the system and and pretty quickly get a sense of like uh here's this weird anomaly

273
00:26:24,580 --> 00:26:30,320
thing happening and at the same time this the the reset password service is kind of acting a little

274
00:26:30,320 --> 00:26:35,840
weird so little wonder that we've got all these calls coming in yeah exactly and i think along with that

275
00:26:35,840 --> 00:26:40,960
if you if you imagine a call center where you've got you know press one for billing press two for customer

276
00:26:40,960 --> 00:26:46,480
support you could instrument each one of those pathways as a separate um separate event basically

277
00:26:46,480 --> 00:26:52,400
and you could say well you know call volume is going up but all of the increase is actually coming from

278
00:26:52,400 --> 00:26:57,520
uh technical support and you can see kind of what pathway people are following and depending on how you set

279
00:26:57,520 --> 00:27:03,360
up your dashboards like maybe that's you you set it up so it's easy to visually see uh which which one of

280
00:27:03,360 --> 00:27:09,040
those is spiking and then kind of decide what to do from there so a kind of a long well a little bit

281
00:27:09,040 --> 00:27:13,760
along those lines as far as like consuming all this stuff i mean one of the more uh i don't know

282
00:27:13,760 --> 00:27:18,240
for exciting as as as my life is and i'm imagining you y'all's as well like one of the more intriguing

283
00:27:18,240 --> 00:27:22,960
parts of the the google sre book was the little i forget what they call it but the query language and the

284
00:27:22,960 --> 00:27:28,960
kind of event um language that they had in there and it looked like a pretty like rich way of expressing

285
00:27:28,960 --> 00:27:34,560
not only querying things but sort of uh uh setting up alerts for events or or whatever you would want

286
00:27:34,560 --> 00:27:39,840
to call that and like so so one that's a fun way to interact with something but when it comes to uh

287
00:27:39,840 --> 00:27:46,000
again uppercase metrics like what's what's like the the interface look like like is it uh is it command

288
00:27:46,000 --> 00:27:51,040
line stuff or a ui or apis or is there some fun little query language like how do people mess around

289
00:27:51,040 --> 00:27:57,680
with it yeah so basically we've tried to keep it as um like let's say usable as possible so we don't

290
00:27:57,680 --> 00:28:03,520
there there's no real command line or query language uh that they use interact with it's essentially all

291
00:28:04,080 --> 00:28:11,120
um adding adding graphs to dashboards and moving them around adjusting time ranges um it's really designed

292
00:28:11,120 --> 00:28:16,000
to be sort of just more of like a drag and drop kind of experience so that there's not really

293
00:28:16,000 --> 00:28:20,560
a query language to learn because because by and large pretty much every time series database has

294
00:28:20,560 --> 00:28:26,880
its own you know little query language or api or whatever it is some of them are much easier than

295
00:28:26,880 --> 00:28:31,760
others some of them like i don't know if you've looked at the elastic search query language but it's

296
00:28:31,760 --> 00:28:37,920
this crazy nested json thing and it's great for computers but it's horrible for people um so there are

297
00:28:37,920 --> 00:28:42,400
and they're you know you could spend six months learning the ins and outs of each one of these

298
00:28:42,400 --> 00:28:48,160
so really we kind of wanted to abstract the database itself and just let users you know look for the

299
00:28:48,160 --> 00:28:53,360
charts they want adjust the arrangement um and make it as quick as possible to get stuff on a dashboard

300
00:28:53,360 --> 00:28:58,400
that you can you can look at and the the thing that we're going to be working on in the next release

301
00:28:58,400 --> 00:29:06,080
which will be 1.5 is going to go more into the uh the alerting side so right now it's basically like

302
00:29:06,080 --> 00:29:10,400
like we were saying it's all visual inspection um but with 1.5 we're going to be adding some

303
00:29:10,400 --> 00:29:15,200
functionality to be able to set uh threshold based alerts and and try and bring in some of that

304
00:29:15,200 --> 00:29:20,880
functionality and i think the goal with that is also going to be to to stay away from making users

305
00:29:20,880 --> 00:29:27,280
have to like learn and write a query language which while that has its its power being able to like

306
00:29:27,280 --> 00:29:33,440
really you know precisely define how these alerts are going to be set up my my hunch and we're still

307
00:29:33,440 --> 00:29:38,000
working with some users to validate this is that most people will get a lot of value out of very

308
00:29:38,000 --> 00:29:45,840
simple alerting um configuration there aren't a ton of people that are going to be going into you know

309
00:29:45,840 --> 00:29:51,360
setting their 45 day moving average and like rolling windows and like all that kind of stuff so we're

310
00:29:51,360 --> 00:29:57,280
going to try and stay away from the things that make really complex user interfaces for alerts i mean the

311
00:29:57,280 --> 00:30:04,160
the sweet spot is the 80 20 point where you can get the things that everyone has to do solved but it

312
00:30:04,160 --> 00:30:09,440
is kind of nice to be able to to drill down if you if you could expose that and i think you're talking

313
00:30:09,440 --> 00:30:15,920
about the the borgmon um time series alerting chapter from the yeah that's sorry there you go

314
00:30:15,920 --> 00:30:23,200
yeah yeah i i love the sre book but it's kind of interesting that they chose to put a bunch of borgmon

315
00:30:23,200 --> 00:30:28,880
configs in a in a chapter that no one could no one could ever use unless they work at google well you

316
00:30:28,880 --> 00:30:34,480
know sometimes you got to fill pages you know like we said this would be 300 the uh the the courier

317
00:30:34,480 --> 00:30:40,720
fixed width stuff sure does fill them out but you know i mean they got it up to 500 so there you go

318
00:30:40,720 --> 00:30:45,520
there you go i'm not but but it is like i said it is interesting to see it and i know you know uh

319
00:30:45,520 --> 00:30:49,520
you know you're you're reminding me that i always love a query language that's sort of like the first

320
00:30:49,520 --> 00:30:55,120
thing you need to understand is a fundamentally inhumane way of laying out data whether it's key value

321
00:30:55,120 --> 00:31:00,560
stores or relational databases and once you understand that this weird language that you use

322
00:31:01,440 --> 00:31:06,080
and then man i can imagine with nested json stuff it would be like writing an xml file just to

323
00:31:06,080 --> 00:31:10,960
like execute a query not that uh everyone in the industry might have done that at some point

324
00:31:10,960 --> 00:31:15,360
but yeah that is uh that is difficult yeah and and i would imagine i mean being kind of sympathetic

325
00:31:15,360 --> 00:31:22,400
to the uh the old 80 20 thing i mean it is uh you know there's always like anti-snowflake

326
00:31:22,400 --> 00:31:26,400
and pets and cattle and stuff like that but as we were talking about earlier when it comes to

327
00:31:26,400 --> 00:31:32,160
actually deeply monitoring your uh your specific applications there probably are some special

328
00:31:32,160 --> 00:31:37,040
snowflakes of how you would want to look at it and query it so it's it's hard to uh it's hard to get

329
00:31:37,040 --> 00:31:42,480
too specialized in a general sense obviously but just having a good baseline of stuff and then i don't

330
00:31:42,480 --> 00:31:47,680
know maybe at some point an advanced tab is uh is always interesting but then there's also you know

331
00:31:47,680 --> 00:31:51,760
to counter counter argument there's also always the thing of like does anyone ever use the advanced

332
00:31:51,760 --> 00:31:56,080
tab and search you just want to like a simple way of searching stuff and then if you have an sdk to

333
00:31:56,080 --> 00:32:00,160
actually instrument and query things on your own that'll probably take care of things

334
00:32:01,120 --> 00:32:06,800
yeah i think longer longer tension go ahead sorry go ahead i was gonna say there's always a tension

335
00:32:06,800 --> 00:32:14,720
between the the usability and that kind of flexible power for any interface but for lots of alerting

336
00:32:14,720 --> 00:32:19,200
and monitoring if you've ever done that kind of work it does get a bit repetitive and if you can make

337
00:32:19,200 --> 00:32:25,280
that super simple and super easy for the 80 case then you're in a good spot and and this also brings

338
00:32:25,280 --> 00:32:29,440
up another point i mean it's it's a bit of a uh i don't know product management sort of business

339
00:32:29,440 --> 00:32:36,880
strategy thing is um i mean it seems like the other answer to this is like well there's a whole bunch

340
00:32:36,880 --> 00:32:41,920
of partner products you can use that integrate with with pivotal cloud foundry and can do their own

341
00:32:41,920 --> 00:32:47,440
sort of analysis of things using you know using uh all of the instrumentation and things like that but

342
00:32:47,440 --> 00:32:52,640
it's not like it's not like you're cut off from other ways of doing things right and that's that's one of

343
00:32:52,640 --> 00:32:57,040
the things i was going to say is that you know we're not we're not going to try and reinvent pager

344
00:32:57,040 --> 00:33:02,880
duty like we're not going to try and take these these products and shove them into uh our product

345
00:33:02,880 --> 00:33:08,400
what the the goal would be if we feel like there's a ton of people using pager duty let's make it as

346
00:33:08,400 --> 00:33:14,960
easy as possible for you to trigger the alert where the data is inside of cloud foundry and then hook

347
00:33:14,960 --> 00:33:20,400
it up to pager duty and make it as easy as possible to sort of like work with you know work with things

348
00:33:20,400 --> 00:33:25,440
that we know are going to be part of this ecosystem um and you know do what we can do best and and make

349
00:33:25,440 --> 00:33:29,680
it as easy as possible for customers to work with the tools that they want to work with so one thing

350
00:33:29,680 --> 00:33:36,720
i'm always interested in uh is is uh especially given given your background is like what's your sense

351
00:33:36,720 --> 00:33:42,560
for like the role that doing things in an open source way plays in in this space and in the metrics

352
00:33:42,560 --> 00:33:47,040
and monitoring and systems management space and and as always forgive me for making a bunch of statements

353
00:33:47,040 --> 00:33:53,840
after asking a question but but it seems to me that like there's not really that many like monitoring

354
00:33:53,840 --> 00:33:59,040
and systems management things that that have been innovated in the past five or so years that aren't

355
00:33:59,040 --> 00:34:03,760
open source i guess i guess there's like sasses you know services that by their very nature

356
00:34:04,480 --> 00:34:09,360
are kind of sort of open source but it seems like a lot of the good work has been going on because

357
00:34:09,360 --> 00:34:13,280
it's an open source and because it has that kind of community around it but i don't know i mean you're

358
00:34:13,280 --> 00:34:18,720
you're this me here so so you tell me is has doing things in an open way helped out in this space a lot

359
00:34:18,720 --> 00:34:24,240
yeah i think i think open source i mean more broadly just in software in general i think we're

360
00:34:24,240 --> 00:34:29,280
seeing a lot more a lot more adoption of things like i mean you look at docker docker was a completely

361
00:34:29,280 --> 00:34:37,360
open source product and um i think companies have become more and more comfortable uh taking on you know

362
00:34:37,360 --> 00:34:42,800
whatever whatever risks they they find associated with open source and licensing and um and that kind

363
00:34:42,800 --> 00:34:49,360
of stuff but i i think for us you know when we started influx data we had originally pivoted from

364
00:34:49,360 --> 00:34:54,720
a closed source product to doing influx db as an open source project because we saw that's kind of where

365
00:34:54,720 --> 00:35:01,840
we saw the i guess the desire for for what we were working on was really in the storage space and we

366
00:35:02,400 --> 00:35:06,720
kind of looked around and saw that there were other other things like elastic search mongo db

367
00:35:07,280 --> 00:35:13,360
that were you know getting a lot of adoption and it really seems like the the biggest reason was

368
00:35:13,360 --> 00:35:19,520
because they were open source products they were really well supported by their their creators they

369
00:35:19,520 --> 00:35:25,760
were moving quickly and it was something that you could invest your time in and not feel like you were

370
00:35:25,760 --> 00:35:30,960
going to have a ton of vendor lock-in within like the the community so if you wanted to go from one job

371
00:35:30,960 --> 00:35:36,720
to another having experience with mongo db was was an asset because you know the likelihood that someone

372
00:35:36,720 --> 00:35:41,680
else was using it was high whereas with some of these other closed source tools like i know there are

373
00:35:41,680 --> 00:35:47,280
tons of java apm tools but i've never worked with most of them and i don't know a lot of people who have

374
00:35:47,280 --> 00:35:52,640
so i feel like you you get a lot of value as an engineer from this sort of open source community

375
00:35:52,640 --> 00:35:58,240
because it means that you can uh specialize in something that's that's popular um and i think

376
00:35:58,240 --> 00:36:05,840
you know with with all this stuff in general i think it's definitely creating uh i guess it's a

377
00:36:05,840 --> 00:36:09,680
double-edged sword because it's creating all these companies that are doing open source things and

378
00:36:09,680 --> 00:36:15,520
these products that are really powerful and and kind of foster a lot of innovation and getting developers

379
00:36:15,520 --> 00:36:23,280
comfortable with them but it's making it hard for people to find uh a business model that will scale

380
00:36:23,280 --> 00:36:29,680
so i think if you look at rethink db as a good example they had a great product rethink db was

381
00:36:29,680 --> 00:36:35,760
really well engineered it had a ton of people who loved it and they just couldn't get traction on the

382
00:36:35,760 --> 00:36:42,560
sales side and get uh companies to actually pay for it and i think that it was it's kind of a shame

383
00:36:42,560 --> 00:36:47,520
because it was such a such a good product and i think at the end stripe ended up picking up a bunch

384
00:36:47,520 --> 00:36:53,280
of the engineers and i think it'll kind of live on um as a tool but it means that open source still

385
00:36:53,280 --> 00:36:58,880
has a little bit of i feel like kind of maturing to do before people will really take it seriously

386
00:36:58,880 --> 00:37:04,640
as a business and i think there are sometimes in in larger organizations uh you get to this decision

387
00:37:04,640 --> 00:37:10,720
point where somebody has to decide to uh spend a lot of money whether it's engineering resources or

388
00:37:10,720 --> 00:37:16,160
licensing fees to to buy into these open source products and they want some sort of a guarantee that

389
00:37:16,880 --> 00:37:20,720
the company is going to be around in a year or two years if they're going to spend spend these

390
00:37:20,720 --> 00:37:26,160
resources so i think it's definitely helping with growth and adoption but i think there's still a lot

391
00:37:26,160 --> 00:37:33,120
a lot to be kind of learned and proven that open source has staying power as a business model so i

392
00:37:33,120 --> 00:37:38,720
spent the last 10 years making a living on open source software and i think there's a there's a couple

393
00:37:38,720 --> 00:37:45,440
things to point out here one is that when people open source it's not really one thing there there's a

394
00:37:45,440 --> 00:37:51,600
spectrum of ways people approach that and and bring that to the you know quote unquote market which is

395
00:37:51,600 --> 00:37:58,960
to say one i think that it's a it's almost a de facto standard now and this is evolving obviously we're

396
00:37:58,960 --> 00:38:05,920
all part of it at pivotal but infrastructure in specific the the standard is open source you know

397
00:38:05,920 --> 00:38:12,560
like the the lower levels the internet's built on linux and it just goes up from there and you know

398
00:38:12,560 --> 00:38:18,080
now we're getting higher and higher up the stack and more and more of these capabilities but but open

399
00:38:18,080 --> 00:38:24,560
sources business model is not a solved problem and at some point you have to set up some way

400
00:38:24,560 --> 00:38:30,640
to have the conversation and and in many cases and i've had a lot of smart friends you know it's very

401
00:38:30,640 --> 00:38:36,640
very smart engineering at least that weren't able to turn the corner at some point you have to have you

402
00:38:36,640 --> 00:38:43,280
have to actually ask for money right like you have to sit across the table and ask someone for money

403
00:38:43,280 --> 00:38:48,240
or they're not going to give you any like that's that so i see a lot of people build lots of value

404
00:38:48,240 --> 00:38:53,520
create lots of value with open source projects and then and then fail to ever ask anyone for money or

405
00:38:53,520 --> 00:39:00,480
even really set up a way someone could pay them yeah i i totally agree with that as well and i think

406
00:39:00,480 --> 00:39:06,560
um i think docker is another good example of you know fantastically popular products and they they

407
00:39:06,560 --> 00:39:12,480
did wait a long time before they really moved towards monetization and i think a lot of other

408
00:39:12,480 --> 00:39:19,920
people kind of saw that they hadn't really seized that that facet of the business and i think they

409
00:39:19,920 --> 00:39:24,240
are facing a lot more competition now as a result because i think people saw that there was

410
00:39:24,240 --> 00:39:28,960
a lot of money being left on the table as a result and kind of went after it and now i think they're

411
00:39:28,960 --> 00:39:36,000
kind of worse off for it nature abhors a vacuum and if you leave empty space then people will fill it

412
00:39:36,000 --> 00:39:40,320
yeah you know i i think i think i think summarizing it a little bit i mean uh uh i always like talking

413
00:39:40,320 --> 00:39:44,880
on that this question asking this sort of question because i think i mean you described it really well

414
00:39:44,880 --> 00:39:50,480
there's this interesting um i don't know like to make a weird visual that's it's like a big like

415
00:39:50,480 --> 00:39:55,120
spring i don't know what the opposite of a sprung sprung spring is but like this coiled up spring and

416
00:39:55,120 --> 00:40:02,080
the tension driving it is like and this is the most mysterious part is just the the raw desire of

417
00:40:02,080 --> 00:40:09,440
developer types to like develop stuff and show it off to people and you know this is a there's a

418
00:40:09,440 --> 00:40:13,440
there's a brief way of putting it which is not entirely accurate but it's you're kind of alluding

419
00:40:13,440 --> 00:40:18,560
to it it's like you want to like be famous and kind of show off which again is not really accurate

420
00:40:18,560 --> 00:40:24,960
like i think a lot more of it is you genuinely want to share like interesting stuff you've done and found

421
00:40:24,960 --> 00:40:29,360
i think you want to solve people want to solve problems they want to solve interesting problems

422
00:40:29,920 --> 00:40:35,040
exactly and so you've got that tension of developers wanting to do that uh and and then the other side

423
00:40:35,040 --> 00:40:39,600
of the the thing the other thing that's feeding into the tension is just like as andrew was saying

424
00:40:39,600 --> 00:40:45,040
like well eventually we're going to need to make money and and like this this like going back and

425
00:40:45,040 --> 00:40:50,560
forth between those two things i mean it's as we were kind of chronically and uh and uh alluding to

426
00:40:50,560 --> 00:40:55,840
it's eaten up a lot of little startups and efforts here but it's also resulted in several like great

427
00:40:55,840 --> 00:41:01,120
ones uh and and ones that in good innovations in this area so while it's kind of mysterious how

428
00:41:01,120 --> 00:41:06,560
that stuff works out i think it's at least in my books it's pretty clear that open source is a good

429
00:41:06,560 --> 00:41:11,520
driver to make innovation happen in the industry both in the business model side and and more in the

430
00:41:11,520 --> 00:41:17,520
technical side but uh yeah it is uh that that that figuring out as andrew was saying like how you

431
00:41:18,080 --> 00:41:24,560
how you harness the desire to solve big problems in the open is is always a bit of a uh some magic how

432
00:41:24,560 --> 00:41:31,680
do you how do you avoid the tragedy of the commons is is really the question exactly exactly well just uh

433
00:41:31,680 --> 00:41:36,160
just one more thing before we wrap up so i mean you alerted to some stuff that's uh that's coming in the

434
00:41:36,160 --> 00:41:42,320
future like uh more attention to um uh alerting and setting up uh things for alerts but when when

435
00:41:42,320 --> 00:41:47,200
you kind of look over the next i don't know i don't know if you live yourself a uh live your life a

436
00:41:47,200 --> 00:41:51,920
quarter a year at a time or whatever but like when you kind of look out six 12 months or so what are

437
00:41:51,920 --> 00:41:57,120
some areas in in uppercase metrics that uh that you think would be interesting to uh work on and get

438
00:41:57,120 --> 00:42:03,920
out there yeah so i think i try to look like one one release out at a minimum uh but i i think the

439
00:42:03,920 --> 00:42:08,480
once we get get past the alerting stuff um and and i think there are some things that are sort of

440
00:42:09,120 --> 00:42:13,680
obvious to most anyone who's looking at a at a metrics and monitoring setup like there are some things

441
00:42:13,680 --> 00:42:17,920
that are just sort of expected and we're trying to fill kind of fill in those gaps and i think

442
00:42:17,920 --> 00:42:23,600
like i said alerting is the first one um there's going to be a little bit more i think to

443
00:42:24,480 --> 00:42:28,880
to allow the platform itself to be slightly more extensible so right now we're kind of

444
00:42:28,880 --> 00:42:34,400
limiting um some of the amount of stuff that people can put in primarily because you know we're also

445
00:42:34,400 --> 00:42:39,520
deploying uh like i said my sql is kind of the storage back end for this we've got some processing

446
00:42:39,520 --> 00:42:44,320
stuff in front of that but basically there is a point where it wouldn't be able to keep up with a

447
00:42:44,320 --> 00:42:48,080
certain amount of metrics and you know we're pretty comfortable with where it's at now but it would be nice

448
00:42:48,080 --> 00:42:54,320
for uh people to be able to open it up a little bit more broadly uh control resolution

449
00:42:54,320 --> 00:43:00,400
you know if you wanted to be able to store less than one minute roll-ups um things like that and

450
00:43:00,400 --> 00:43:06,160
those are kind of just like nice to haves but i think the the bigger thing that i have on my mind

451
00:43:06,160 --> 00:43:13,200
and i i haven't really this isn't necessarily uh shared too broadly yet but it's kind of i see that

452
00:43:13,200 --> 00:43:17,760
there are a lot of places within the cloud foundry ecosystem where we're kind of leaving a lot of really

453
00:43:18,480 --> 00:43:24,160
interesting data uh to to just sort of it's just going nowhere right now so i think there are a lot

454
00:43:24,160 --> 00:43:30,480
of lower level things where i think we can pull um some container networking metrics there are lots of

455
00:43:30,480 --> 00:43:35,520
things where i think we've got a lot of data within the system that we can start to take advantage of

456
00:43:35,520 --> 00:43:42,000
and we can start to build really specialized tools and visualizations to kind of look at you know if one

457
00:43:42,000 --> 00:43:46,960
of your let's say one of your nodes gets compromised and somebody's sending a ton of data out over the

458
00:43:46,960 --> 00:43:51,840
network you might see a spike on the the outbound networking port and things like that that might

459
00:43:51,840 --> 00:43:56,720
be really easy to look at if you're looking at a like a network topology graph but maybe not so easy

460
00:43:56,720 --> 00:44:01,120
to look at if you're looking at you know aggregate network traffic so i think there are lots of things

461
00:44:01,120 --> 00:44:07,200
like that where we can start to build tools that make you know cloud foundry specifically much more

462
00:44:07,200 --> 00:44:12,720
powerful and make the metrics that we're already uh emitting way more valuable to the end users and

463
00:44:12,720 --> 00:44:17,280
that's sort of where i feel like the sweet spot of uh it's a combination of our metrics team and the

464
00:44:17,280 --> 00:44:21,600
platform metrics team i think once we can start to harness a lot of the data that we already have

465
00:44:22,960 --> 00:44:29,760
there will be a ton of value for the customers and the operators in just kind of using cloud foundry

466
00:44:29,760 --> 00:44:33,360
better and i think that's that's kind of where i feel like the next six to 12 months are going to be

467
00:44:33,920 --> 00:44:40,080
the most interesting is when we actually start to to make the entire system better for the end user

468
00:44:40,080 --> 00:44:45,680
yeah it's it's it's uh put it putting uh putting more smi in as it were yeah like fine putting more

469
00:44:45,680 --> 00:44:51,440
smi into the 80 of commonality and and exposing all of that exactly i thought it'd be really cool to

470
00:44:51,440 --> 00:44:58,160
be able to expose some of the wash level platform infrastructure metrics to be able to compare the

471
00:44:58,160 --> 00:45:04,240
the behavior of infrastructure services i think that we have enough meaningful interaction with a lot of

472
00:45:04,240 --> 00:45:10,560
the infrastructure services that we could uh we could do some some really interesting really interesting

473
00:45:10,560 --> 00:45:16,800
analysis of how those perform relative to each other yeah and i think i i know the um like i said the

474
00:45:16,800 --> 00:45:21,680
plot metrics team i know they're looking at a lot of a lot of those kinds of things that we can we

475
00:45:21,680 --> 00:45:27,120
can start to extract that will make it very easy as an operator to just sort of look at a screen i mean

476
00:45:27,120 --> 00:45:32,720
initially look at a screen and just say you know red or green is this thing healthy is this thing you

477
00:45:32,720 --> 00:45:37,040
know going down and then you know the next step kind of like what we're talking about adding for

478
00:45:37,040 --> 00:45:42,720
application metrics they'll i'm sure add some sort of um automated alerting that if you see something

479
00:45:42,720 --> 00:45:49,040
within the platform go down you don't have to to wait to check it you get alerted um either as

480
00:45:49,040 --> 00:45:54,480
it's happening or as soon as it happens well thanks for being on that was good stuff no problem thank you

481
00:45:54,480 --> 00:45:59,600
for having me it's always it's always fun to go down a metrics and monitoring memory lane with myself

482
00:45:59,600 --> 00:46:05,760
like i was just as as you're going over extending the uh the raw data if you will over time i was i

483
00:46:05,760 --> 00:46:11,120
was remembering i after working in this space for like five years way back when i sort of me and my

484
00:46:11,120 --> 00:46:16,400
friends established this principle that like around release three of your systems management project you

485
00:46:16,400 --> 00:46:22,080
basically spend all your time managing the database and and figure figuring out how to how to get all

486
00:46:22,080 --> 00:46:26,400
that data in there and i mean it goes all the way back to the beginning about how how advances in

487
00:46:26,400 --> 00:46:32,080
managing time series data like opens up like uh it frees up a lot of time because uh it's always going

488
00:46:32,080 --> 00:46:36,720
to come managing that in a database is difficult it's a problem with the ingest problem with the queries

489
00:46:36,720 --> 00:46:42,160
or problem with the storage that's right that's right who knew the problem with computers would be computers

490
00:46:42,160 --> 00:46:49,360
uh well uh so i just have a few things i want to call out here at the end uh so we have uh we have

491
00:46:49,360 --> 00:46:54,000
got a lot of fun conferences going on uh one of them in particular we just announced uh this coming

492
00:46:54,000 --> 00:46:58,320
out i'm going to be working at this and someone else on andrew and i's uh team mark heckler will be

493
00:46:58,320 --> 00:47:04,640
working on it so we had a request in the uh the washington dc area to do a slightly uh government uh

494
00:47:04,640 --> 00:47:10,160
oriented um uh one of our little workshops so we're going to be doing that on uh june 7th i'll put a

495
00:47:10,160 --> 00:47:13,600
link in the show notes to it but if you're interested in it's it's i think it's like two

496
00:47:13,600 --> 00:47:19,120
thirds of a day and uh it's free of course you can come and just hear about uh a lot of the the sort

497
00:47:19,120 --> 00:47:25,040
of pivotal way of doing things uh and also we have uh in chicago we have spring days coming up if you

498
00:47:25,040 --> 00:47:31,920
go to springdays.io you can see that and that's may 30th and uh not as my notes say 21st it's probably

499
00:47:31,920 --> 00:47:37,920
may 30th and 31st uh and there'll also be one of those in new york and atlanta uh that you can you

500
00:47:37,920 --> 00:47:43,200
can look up there but that's uh two days of as you would imagine if you're into microservices or

501
00:47:43,200 --> 00:47:48,080
development or java or spring or uh or you know just looking not to have to go to your real job

502
00:47:48,080 --> 00:47:52,800
it's uh it's a good way to spend those two days of time in chicago new york and atlanta and then

503
00:47:52,800 --> 00:47:58,640
finally there's the cf summit the cloud foundry summit uh which goes over how large organizations

504
00:47:58,640 --> 00:48:02,800
are doing all their cloud native digital transformation and also the technical side

505
00:48:03,360 --> 00:48:09,440
of uh of doing things in a cloud foundry way all the way down to the bottom of the application stop

506
00:48:09,440 --> 00:48:16,480
stack up to the top uh more spring oriented and application oriented so that's june 13th to 15th

507
00:48:16,480 --> 00:48:20,640
and i'll put this in the show notes as well or you can get out your pencil and write this down but if

508
00:48:20,640 --> 00:48:28,880
you want to get 20 off of that you can use the code cfsv17 Coté Coté that's my name uh and then you know

509
00:48:28,880 --> 00:48:33,840
later in the year the SpringOne platform but that's that's way far ahead planning uh so with

510
00:48:33,840 --> 00:48:38,240
that thanks for listening as always this has been Pivotal Conversations if you want to go look at

511
00:48:38,240 --> 00:48:45,040
the archives and and find uh a way to subscribe find the rss feed and all kinds of other doodaddles

512
00:48:45,040 --> 00:48:50,080
uh you can go to soundcloud.com slash Pivotal Conversations with no space we also post the full

513
00:48:50,080 --> 00:48:57,360
show notes on our blog at pod not podcast at pivotal.io slash podcast and uh i'm Coté in twitter

514
00:48:57,360 --> 00:49:02,160
if you want to follow up with me that's c-o-t-e how about yourself guest where can people scurry

515
00:49:02,160 --> 00:49:07,840
around with you on the internet uh i i don't tweet much but um i guess i i do have a twitter account

516
00:49:07,840 --> 00:49:13,440
it's just at todd pearson t-o-d-d-p-e-r-s-e-n i'll i'll tweet something eventually

517
00:49:13,440 --> 00:49:20,000
that's right and yourself andrew i'm little idea on twitter fastest way to get my attention

518
00:49:20,000 --> 00:49:23,680
so uh with that we'll see everyone next time bye bye