Cloud-Native Monitoring & PCF Metrics, With Todd Persen (Ep. 65)

Episode 52 · 49:23 · SRT · archive.org · all Tanzu Talk

Well this week Richard can make it so we have uh I don't know perhaps the uh the the second most second or third most popular person aside from me on this podcast yeah this is Andrew Andrew Clay Schaefer we're gonna I'm I'm standing in that's right I I think I think uh it'll be great I answered the call that's right yeah I I just uh it occurred to me that uh because Richard was going to be unavailable for a couple weeks I should get someone else to co-host because despite my self-proclaimed popularity it's kind of like salt like too much and it ruins it so we gotta we gotta balance it out with something uh so what's what's what's the spice of mine uh sugar is that is that right I am definitely not sugar have you talked to me maybe maybe you're uh uh I don't know what do they got over there uh some sort of like uh southwestern uh chili yeah are you some of those are you a mixture of chili and garlic on a string left out to dry in the desert probably that's moving on well uh there you go with that tone setting so we have a guest on this week why don't you introduce yourself sure uh my name is Todd Pearson um I was uh a labs pivot back in New York many years ago uh and departed to start a company that became uh influx data so we were the creators of the open source time series database influx db um and I did that for the last five years or so and then returned to pivotal um primarily on the engineering side but my uh sort of subtitle I guess if you could call it is uh subject matter expert for metrics um effect effectively that just means I'm kind of working cross team to help um sort of bring metrics to the forefront and help with best practices and kind of make sure that we have

some sort of an overarching theme that we're adhering to as far as metrics go across all of cloud found tree so that was before sorry just started in before we before we get into metrics I had a few at a few uh what's the opposite of a follow-on question a lead-on question one uh so uh we got leading the witness there you go so so we had a uh uh this is kind of like a follow-up to the uh nice Andrew's on to the google sre book uh book review uh that we did because there is there's quite a bit about uh not only like uh monitoring and metrics but also time series data or data however you want to say it I always suffer from growing up watching star treks the next generation so I always say it wrong but uh anyhow which leads me to my second thing now this is a little bit of like uh you know yokel Coté asking this but since you worked on some time series data stuff and I think Andrew and I discussed this back back a while I'm curious about your take of like so what's what's the deal with like all the hype around time series data stuff like is it actually like something brand new that hasn't existed before or is it just like oh this is a really cool thing that we should all be using more and as context so back when I did real work and I programmed like I worked at BMC on a BMC performance manager a gloriously named product and you know we did monitoring but uh back then people wouldn't really use the phrase time series data but it seems to have been like it's like a big deal nowadays so maybe uh I just you know went off and learned how to do powerpoint and stop paying attention and people have always talked about that but it seems like it's a phrase that happens a lot more than uh than I'm used to yeah it's definitely been something that has kind of entered the I don't know popular vernacular more in the in the last few years and it was something that we struggled with a bit at at influx data because um not everyone really understands what it is yet um you tell them

metrics and analytics and they kind of get a sense for it but time series data as a larger category still seems a little bit vague and uh generic to a lot of people but really it's it's not all that new it's it's primarily been in the financial arenas it's if you talk to economists and things that they're much more familiar with time series data because they've been using it for um you know forecasting and analysis for you know decades at least um and a lot of uh the earlier work in in time series databases was for financial analysis so if you're thinking about um stock trades and things like that where you have a high volume of transactions that have to be stored and processed those are those were some of the first time series databases that really were were popularized in software um but I would say that there's definitely been a a growth over the last 10 years or so in kind of modern software development and how how people think about time series data and one of the I think one of the things that really pushed pushed it into our I guess point of view and um I guess larger like software development in in general was uh you know new relic became something that people could just sort of drop into the the ruby on rails application with a single a single library and you had a lot of people who were relatively novice web developers that were kind of like growing up with rails that suddenly had these powerful tools that they could get um really easily and it that sort of became a much more common with the rise of you know sass monitoring tools but um lucerne the guy that started new relic had a company before new relic called wiley technologies he sold to um i think computer associates you know several years before starting new relic and that was also the same type of you know performance monitoring metrics collection but it was all for the enterprise java stack so it's been around it's just that now the availability of tools like new relic and the popularization of things like graphite which was you know written in python and uh used heavily at companies like etsy

having those things kind of floating around and establish more of these best practices has really made them something that i think people are feeling much more comfortable with and starting to expect rather than uh kind of just like an experimental thing that people are using to to test out their applications i know that that explanation that that actually makes a lot of sense and kind of comports with my foggy memories of uh i mean tell me if this summary is wrong is uh uh as you're saying we just have much more uh powerful tools and just raw computation and storage that we can apply to it so you can do a lot more with with time-based series than uh back when i was worrying about mibs and stuff like that which uh certainly maps to what seems to be available out there anyone doing monitoring was always storing time series data i think that the thing that's changed is people we have the dark ages of you know the sql databases which you can also use as queues by the way but when you think about the time series databases and some of the stuff that's out there with influx or imgb or even cassandra for that matter like you you can think about the queries and the and the questions you want to ask and they're optimized for for asking and answering those kind of queries about you know the trends in those in those functions that are stored as time series instead of whatever relational algebra right yeah that meant yeah and one thing i was going to point out is that um you brought up relational databases kind of in the in the scope of time series databases and most of new relic for the first several years at least was powered entirely by mysql it was just like a massively sharded mysql setup that that's how that's how everyone starts yeah that's how it starts because it works i mean it's it's well known and i think a lot of the custom time series databases have been a much more like fancy modern thing and there's there's a need for it but the need comes much later when you start to scale i guess i guess mysql is the uh the trash 80 of this generation of systems management people it's it's like what everyone grew up on yeah i always like

stories of the legendary wiley that was always a uh a fun company back then excellently named as well uh also all sorts of meanings well then that brings us to uh to to the nominal topic at hand so uh like you're saying you you started off uh as as a you know working in pivot labs for a while went off to uh explore what we were just talking about and then you came back to be one of my favorite acronyms a smee i think i think that might have been one of the rejected names for smurfs when they were thinking that up but to a subject matter expert all around and so so first of all describe what metrics is like kind of what uh what it does and what uh what need it fills in you know that sort of context for it sure so um specifically are you asking about pcf metrics that sort of product absolutely uppercase metrics uppercase metrics another way to frame it is now we have a time series database what should we do with it well surprisingly or not so surprisingly our time series database is still mysql under the under the hood for now uh because it it does what we need so um what what pcf metrics is trying to do right now um and i'll i'll explain like kind of where we are and where metrics as like a broader concept is headed inside of cloud foundry but the pcf metrics team currently is focusing on uh was initially focused mostly on instance level metrics kind of pulling cpu memory disc stats from um cloud foundry instances and the new release that's coming out uh probably somewhere around the same time as 1.11 uh is going to allow application developers to easily emit metrics from within their code so um there's some some hooks that we've got uh in spring boot uh through using drop wizard metrics as well um that allow application developers to easily wrap endpoints um just wrap blocks of code with timers and things like that and those will all automatically get uh sent

over to a metrics api that we were publishing and then that will get pulled into the pcf metrics application and from there you can build custom dashboards and set up charts to look at metrics the way that you want similar to what you do with datadog or something like that but it's all sort of automatically configured within the cloud foundry infrastructure so so there there's a little aspect just to uh just to call it out that that uh i don't know it's subtle but important and andrew and i were talking about this when we were talking about the sre book is the uh the notion that a developer could actually instrument their application is is sort of phenomenal it's almost an advance in uh in in the multi-decade journey of systems management and i and i think uh especially if you're kind of angling towards those weekly or daily releases that that we try to get people towards with the whole uh pivotal approach to things that uh i don't know i mean that just opens up so many things that that are nice like previously you would have to treat um uh you know the the resource that you are monitoring and managing as a black box but now with like the built-in notion that you can just add in i guess not whatever you want but more or less whatever you want seems like a huge benefit and then and then also it drives kind of a change to how you think about i i would imagine coding up something like uh all the metric stuff to make it more developer friendly yeah and i think it's it's something that um i think has been a little bit more common like if you look at datadog they've been they've been on a bit of a tear for the last few years because like you're saying that concept of being able to allow developers to sort of fire metrics off into a service and have them get collected and then make it easy to surface those is is really appealing and it does it does make it a lot easier to kind of you know you you may have code that's functional but you may not be able to without load see how it's going to behave um once real users start hitting it so i think being able to have those things in place um is is really powerful

as a as a developer and as a team when you want to start troubleshooting and the nice thing about having it you know easily deployable inside of cloud foundry with basically no configuration is that you don't have to sit there and watch your datadog bill as users just start sending more and more stuff because that it has a tendency to add up rather quickly um and this is a nice way for people to kind of bridge that gap between having something that's can be prohibitively expensive at a certain point um versus having something that you can basically get out of the box for for free well it's it's just part of the value of the platform but this this observability this kind of telemetry is critical no one's going to have a better idea or understanding of the app's health and the developers who wrote it in the first place so it's the best it's the best time to add that yeah and i think if you can get um you get your teams thinking that way and thinking about the instrumentation as almost almost part of a documentation slash testing kind of mindset i think it can go a long way in helping the the reliability of apps especially when you're deploying them inside of a cloud infrastructure and you may not necessarily be able to as easily you know watch them and and monitor them because they're inside of containers or whatever they're not quite as accessible from the outside one one magic way to give them that mindset is to make them wear pagers for their app that's true then they'll monitor nothing on on the topic of you know on in the vein of being a subject matter expert so first um someone who would be you know just taking care of the monitoring and using something like uppercase metrics like in your mind first like what are the kind of like personas or people who get involved in that and then and then as you delve into that like what are what are like the things they would want to look at and obviously like uh this is dependent on the exact type of application but in general like what are the kind of things they would want to instrument and kind of how they would want to interact with and query this data like what's

what's your view of what the user would like to do as they say sure yeah and i i think um on the on the topic of personas i feel like so there's there's another sort of part to this whole thing which is the project that we're sort of working on right now that's you know being dubbed pcf metrics is sort of like half of what's really going on and there's also another piece that is going to be platform metrics which will be is being built by a separate team it will live as a separate uh product for now but the idea is that that sort of covers more of the things that tell you about the health of your cloud foundry installation are your services up and running so there are things like that that are geared more towards the the operators as a persona whereas the pcf metrics app that i'm working on with with this team is more geared towards sort of developers and developers handing things off to whoever might be on on call so it could be somebody who's part of the team like here we sort of go through a rotation where developers take on call shifts um for for things on pws but if you have a dedicated ops or sre team that's kind of looking at these metrics for for different services or have some sort of uh kpis that they want to make sure that they're hitting um that's sort of the the two halves to the the real i think user persona of this pcf metrics and generally a lot of the things that you're looking at like you said will be sort of application dependent but there are always the sort of soft ones like cpu utilization memory utilization those are kind of like it could be high it could be low depends on if you know how you're using your resources if that's correct or not like if you're using redis maybe the memory is always kind of high or maybe you're always cpu bound um but those are those are the things that some people look at as just sort of a kind of a sniff test for whether or not things look like they're under or over provisioned and then i think the things that really have the most value are the ones that we're kind of moving toward towards in this next release where you can say um how many items are in my redis queue or what's the approximate time

taken to process a job and there are things that you can sort of uh keep an eye on that are sort of part of either like a processing pipeline or sort of part of your app's overall response cycle that can tell you really quickly like oh the queue depth has been at 100 for the last week and all of a sudden it's 4 000 something's not getting processed correctly or somebody's slowing down it could network could be a crashed application and those are the things that i think starting to be able to get sort of customized views into uh are really where you can start to get a lot of value out of out of metrics um so you sort of you sort of got the uh uh checking in on the behavior of the systems you're looking at right like so some of the things you're listing were like uh uh what's the flow of of data or processes through a pipe or whatever and just kind of making sure that uh it kind of looks normal i guess or or even when it's not looking normal to kind of check in and see what's going on with it and and kind of looking at that and then and then there's there's the uh the second thing you it's kind of the other side of that which is like uh trying to automate the detection of a weird state like like you were saying you know normally we would expect this result to be like 100 but now it's 10 000 so maybe that's great or maybe that's terrible and and we should probably figure out what's going on with that right and i think the the the you sort of said something about sort of automatic detection um and i feel like there's sort of this holy grail that everybody wants which is sort of uh automatic anomaly detection of error condition in in anything and that's something that's pretty far off just simply because it's it's so much data to process so there are a few things that people have done to do sort of statistical correlations between different time series and some of that's proved useful but um one thing that etsy sort of pioneered or or at least like logged about extensively was just pure visual inspection you get your dashboards and you you know what they're supposed to look like and you look at them all the time and as soon as you see something

that looks you know out of whack like and i know especially at etsy they deploy it all the time and every time they do a deployment engineers would just stare at their dashboards and they'd look for a spike or a drop or something that looked like it was out of out of what they would expect based on just a visual observation and i think that that actually could be really powerful um just for that that quick observation uh that something's going wrong human vision is pretty good at detecting anomalies so there's there's a bunch of conversations that are going along in the industry and i think pinnable is in a unique place to help people understand um or or you know progress with respect to things like the service level objectives i don't know if you've if you've had a chance to read the their sre book or heard some of the conversations about some of the cre work that we're doing with google right now but but being able to take and frame an application's behavior in in terms of service level objectives and then service level indicators is is game changing and it's especially it's especially lacking in kind of that enterprise segment and in the conversations i have um you know with a lot with a lot of our customers they're they're they're actually they don't know what their objectives are they don't know what their indicators are and so being able to help people see that clearly frame that clearly with a shared metaphor across their organization and then and then also coming back into our product i think that will that'll help advance and accelerate a lot of the the benefits that people are getting out of the platform i completely agree with that and i feel like the i think the google sre book is sort of um i guess in the way that i was saying that new relic sort of made people start to uh realize that they could come to expect you know being able to have metrics as sort of a a tool that was always available to them as a developer i feel like the the sre book has sort of pushed this um this concept of services monitoring into

much more of a mainstream topic i've seen it come up at a lot of different companies now where you know they're doing a book club they're going through the sre book and people are getting more familiar with with slos and i think having having the technology and infrastructure to be able to to instrument your applications made it possible to start having those conversations around what are the things that we want to look at and how do we how do we define those goals and now it seems like maybe the the trickiest part in some of these cases is if you're talking about uptime how do you measure the uptime like now we know how to record it we know how to visualize it but what what thing in your code do you instrument to like determine yeah with some sort of reliability is the uptime x like how do i get that out so i think it's giving people a way to start uh experimenting with that stuff and really uh making it it possible and i think you're right that i think pcf metrics does give teams you know within cloud foundry an easy way to just have this place to start experimenting with those kinds of of goals i i think the uptime is is one thing that is maybe nebulous but you can always figure out what your business does and monitor that right like we we sell this thing if how many did we sell the last minute how many did we sell or whatever our transactions whatever our meaningful metrics are and they're going to be contextual contextual to every organization but i think measuring those high level things often gives you more more insight than than the rest of the system and maybe you want to drill down when you're troubleshooting but the only thing that matters for your business are did you sell the thing not not the rest of the numbers so so along those lines i'm curious i mean we'll see we'll see if this works out but as pulling all that together including what andrew was saying like if uh i'm remembering uh there's there's one talk that uh who was it uh a comcast person gave at spring one platform james taylor but not the james taylor uh and uh he was saying you know one of the things

they wanted to fix was preventing people from uh calling into a call center because that basically is five dollars off the off the top right there and then uh and then the costs just escalate as you're talking to a human on the phone and so they were putting in place a uh basically uh well i forget the exact application but it was something to uh help people uh figure out what they would do instead of calling into a call center i think you know techs and in customers of comcast and so there's an interesting little metric there of like we would like to monitor how many people are calling into the call center like because we want that number to be lower and so um how how like if someone were like coding up their own instrumentation for an application like how would they instrument that business data if you will into it like how would that surface and kind of get integrated into the overall health of the service they were doing so i think in the most basic sense um you know some somewhere in their code they've got this concept of uh you know a call being received and you know there are a couple like different types of things that you can collect some things are uh like a metric which is basically like something that has a value at a given point in time so that could be like cpu utilization memory utilization it's something that's sort of got just a number that varies over time and then the other side of it is you can also just have things that are more like events so a call being received is an event and it doesn't really have a value associated with it it's sort of like a unit of one um and you can just send send those events and then have something on the back end that's that's receiving them or something that's either like uh this is part of the way that some of the stuff in spring boot works you've got like a little accumulator basically that will keep track of how many times you increment a counter and then when you uh flush it once a minute or whatever it'll send the the count that was received in the past minute and essentially that gets pushed out to then that becomes like a metric so then it's like how many calls per minute did i get or whatever and then that sort of gets uh sent out to you know our time series database effectively

and with that then you can suddenly have something that you're able to chart uh over you know over time you initially start looking at it per minute maybe you're doing an aggregation and you want to know what is it per hour what is it per week what was it this week versus last week and once you start collecting that data now you've got this sort of powerful trending that you can do and say like am i am i decreasing my overall call center volume month over month and you can you can set that stuff up and right now we're kind of focusing on two week retention periods for pcf metrics but the the goal is be able to open that up so that you know for some metrics you could store 30 days or 100 days or whatever it is that you're looking over and then you can make that sort of a business goal looking at month over month what is what is the the increase or decrease look like so so essentially it's it's the uh same old integration thing is like well somewhere there needs to be a uh a unit of compute whether it's uh whether it's a a fun old script or something more sophisticated that's firing off an event or somehow noting that like a call has come in or whatever and then you're essentially just aggregating that into some point and then another another little uh part of the system comes along and does something with that that data if only just displaying it or providing it to somewhere and then and then i would imagine i mean i guess you could think of it as the uh i mean the classic scenarios i remember from apm are always uh buying a book and like forget password or whatever yep and uh you could sort of monitor that your automated forget password system uh was was probably going screwy if all of a sudden you had a huge spike in people calling in to uh reset their password i mean that's a really simplistic thing but it it does it does seem like one of those things where you uh you could pull together a lot of different parts of the system and and pretty quickly get a sense of like uh here's this weird anomaly thing happening and at the same time this the the reset password service is kind of acting a little weird so little wonder that we've got all these calls coming in yeah exactly and i think along with that

if you if you imagine a call center where you've got you know press one for billing press two for customer support you could instrument each one of those pathways as a separate um separate event basically and you could say well you know call volume is going up but all of the increase is actually coming from uh technical support and you can see kind of what pathway people are following and depending on how you set up your dashboards like maybe that's you you set it up so it's easy to visually see uh which which one of those is spiking and then kind of decide what to do from there so a kind of a long well a little bit along those lines as far as like consuming all this stuff i mean one of the more uh i don't know for exciting as as as my life is and i'm imagining you y'all's as well like one of the more intriguing parts of the the google sre book was the little i forget what they call it but the query language and the kind of event um language that they had in there and it looked like a pretty like rich way of expressing not only querying things but sort of uh uh setting up alerts for events or or whatever you would want to call that and like so so one that's a fun way to interact with something but when it comes to uh again uppercase metrics like what's what's like the the interface look like like is it uh is it command line stuff or a ui or apis or is there some fun little query language like how do people mess around with it yeah so basically we've tried to keep it as um like let's say usable as possible so we don't there there's no real command line or query language uh that they use interact with it's essentially all um adding adding graphs to dashboards and moving them around adjusting time ranges um it's really designed to be sort of just more of like a drag and drop kind of experience so that there's not really a query language to learn because because by and large pretty much every time series database has its own you know little query language or api or whatever it is some of them are much easier than others some of them like i don't know if you've looked at the elastic search query language but it's this crazy nested json thing and it's great for computers but it's horrible for people um so there are

and they're you know you could spend six months learning the ins and outs of each one of these so really we kind of wanted to abstract the database itself and just let users you know look for the charts they want adjust the arrangement um and make it as quick as possible to get stuff on a dashboard that you can you can look at and the the thing that we're going to be working on in the next release which will be 1.5 is going to go more into the uh the alerting side so right now it's basically like like we were saying it's all visual inspection um but with 1.5 we're going to be adding some functionality to be able to set uh threshold based alerts and and try and bring in some of that functionality and i think the goal with that is also going to be to to stay away from making users have to like learn and write a query language which while that has its its power being able to like really you know precisely define how these alerts are going to be set up my my hunch and we're still working with some users to validate this is that most people will get a lot of value out of very simple alerting um configuration there aren't a ton of people that are going to be going into you know setting their 45 day moving average and like rolling windows and like all that kind of stuff so we're going to try and stay away from the things that make really complex user interfaces for alerts i mean the the sweet spot is the 80 20 point where you can get the things that everyone has to do solved but it is kind of nice to be able to to drill down if you if you could expose that and i think you're talking about the the borgmon um time series alerting chapter from the yeah that's sorry there you go yeah yeah i i love the sre book but it's kind of interesting that they chose to put a bunch of borgmon configs in a in a chapter that no one could no one could ever use unless they work at google well you know sometimes you got to fill pages you know like we said this would be 300 the uh the the courier fixed width stuff sure does fill them out but you know i mean they got it up to 500 so there you go

there you go i'm not but but it is like i said it is interesting to see it and i know you know uh you know you're you're reminding me that i always love a query language that's sort of like the first thing you need to understand is a fundamentally inhumane way of laying out data whether it's key value stores or relational databases and once you understand that this weird language that you use and then man i can imagine with nested json stuff it would be like writing an xml file just to like execute a query not that uh everyone in the industry might have done that at some point but yeah that is uh that is difficult yeah and and i would imagine i mean being kind of sympathetic to the uh the old 80 20 thing i mean it is uh you know there's always like anti-snowflake and pets and cattle and stuff like that but as we were talking about earlier when it comes to actually deeply monitoring your uh your specific applications there probably are some special snowflakes of how you would want to look at it and query it so it's it's hard to uh it's hard to get too specialized in a general sense obviously but just having a good baseline of stuff and then i don't know maybe at some point an advanced tab is uh is always interesting but then there's also you know to counter counter argument there's also always the thing of like does anyone ever use the advanced tab and search you just want to like a simple way of searching stuff and then if you have an sdk to actually instrument and query things on your own that'll probably take care of things yeah i think longer longer tension go ahead sorry go ahead i was gonna say there's always a tension between the the usability and that kind of flexible power for any interface but for lots of alerting and monitoring if you've ever done that kind of work it does get a bit repetitive and if you can make that super simple and super easy for the 80 case then you're in a good spot and and this also brings up another point i mean it's it's a bit of a uh i don't know product management sort of business strategy thing is um i mean it seems like the other answer to this is like well there's a whole bunch of partner products you can use that integrate with with pivotal cloud foundry and can do their own

sort of analysis of things using you know using uh all of the instrumentation and things like that but it's not like it's not like you're cut off from other ways of doing things right and that's that's one of the things i was going to say is that you know we're not we're not going to try and reinvent pager duty like we're not going to try and take these these products and shove them into uh our product what the the goal would be if we feel like there's a ton of people using pager duty let's make it as easy as possible for you to trigger the alert where the data is inside of cloud foundry and then hook it up to pager duty and make it as easy as possible to sort of like work with you know work with things that we know are going to be part of this ecosystem um and you know do what we can do best and and make it as easy as possible for customers to work with the tools that they want to work with so one thing i'm always interested in uh is is uh especially given given your background is like what's your sense for like the role that doing things in an open source way plays in in this space and in the metrics and monitoring and systems management space and and as always forgive me for making a bunch of statements after asking a question but but it seems to me that like there's not really that many like monitoring and systems management things that that have been innovated in the past five or so years that aren't open source i guess i guess there's like sasses you know services that by their very nature are kind of sort of open source but it seems like a lot of the good work has been going on because it's an open source and because it has that kind of community around it but i don't know i mean you're you're this me here so so you tell me is has doing things in an open way helped out in this space a lot yeah i think i think open source i mean more broadly just in software in general i think we're seeing a lot more a lot more adoption of things like i mean you look at docker docker was a completely open source product and um i think companies have become more and more comfortable uh taking on you know whatever whatever risks they they find associated with open source and licensing and um and that kind

of stuff but i i think for us you know when we started influx data we had originally pivoted from a closed source product to doing influx db as an open source project because we saw that's kind of where we saw the i guess the desire for for what we were working on was really in the storage space and we kind of looked around and saw that there were other other things like elastic search mongo db that were you know getting a lot of adoption and it really seems like the the biggest reason was because they were open source products they were really well supported by their their creators they were moving quickly and it was something that you could invest your time in and not feel like you were going to have a ton of vendor lock-in within like the the community so if you wanted to go from one job to another having experience with mongo db was was an asset because you know the likelihood that someone else was using it was high whereas with some of these other closed source tools like i know there are tons of java apm tools but i've never worked with most of them and i don't know a lot of people who have so i feel like you you get a lot of value as an engineer from this sort of open source community because it means that you can uh specialize in something that's that's popular um and i think you know with with all this stuff in general i think it's definitely creating uh i guess it's a double-edged sword because it's creating all these companies that are doing open source things and these products that are really powerful and and kind of foster a lot of innovation and getting developers comfortable with them but it's making it hard for people to find uh a business model that will scale so i think if you look at rethink db as a good example they had a great product rethink db was really well engineered it had a ton of people who loved it and they just couldn't get traction on the sales side and get uh companies to actually pay for it and i think that it was it's kind of a shame

because it was such a such a good product and i think at the end stripe ended up picking up a bunch of the engineers and i think it'll kind of live on um as a tool but it means that open source still has a little bit of i feel like kind of maturing to do before people will really take it seriously as a business and i think there are sometimes in in larger organizations uh you get to this decision point where somebody has to decide to uh spend a lot of money whether it's engineering resources or licensing fees to to buy into these open source products and they want some sort of a guarantee that the company is going to be around in a year or two years if they're going to spend spend these resources so i think it's definitely helping with growth and adoption but i think there's still a lot a lot to be kind of learned and proven that open source has staying power as a business model so i spent the last 10 years making a living on open source software and i think there's a there's a couple things to point out here one is that when people open source it's not really one thing there there's a spectrum of ways people approach that and and bring that to the you know quote unquote market which is to say one i think that it's a it's almost a de facto standard now and this is evolving obviously we're all part of it at pivotal but infrastructure in specific the the standard is open source you know like the the lower levels the internet's built on linux and it just goes up from there and you know now we're getting higher and higher up the stack and more and more of these capabilities but but open sources business model is not a solved problem and at some point you have to set up some way to have the conversation and and in many cases and i've had a lot of smart friends you know it's very very smart engineering at least that weren't able to turn the corner at some point you have to have you have to actually ask for money right like you have to sit across the table and ask someone for money

or they're not going to give you any like that's that so i see a lot of people build lots of value create lots of value with open source projects and then and then fail to ever ask anyone for money or even really set up a way someone could pay them yeah i i totally agree with that as well and i think um i think docker is another good example of you know fantastically popular products and they they did wait a long time before they really moved towards monetization and i think a lot of other people kind of saw that they hadn't really seized that that facet of the business and i think they are facing a lot more competition now as a result because i think people saw that there was a lot of money being left on the table as a result and kind of went after it and now i think they're kind of worse off for it nature abhors a vacuum and if you leave empty space then people will fill it yeah you know i i think i think i think summarizing it a little bit i mean uh uh i always like talking on that this question asking this sort of question because i think i mean you described it really well there's this interesting um i don't know like to make a weird visual that's it's like a big like spring i don't know what the opposite of a sprung sprung spring is but like this coiled up spring and the tension driving it is like and this is the most mysterious part is just the the raw desire of developer types to like develop stuff and show it off to people and you know this is a there's a there's a brief way of putting it which is not entirely accurate but it's you're kind of alluding to it it's like you want to like be famous and kind of show off which again is not really accurate like i think a lot more of it is you genuinely want to share like interesting stuff you've done and found i think you want to solve people want to solve problems they want to solve interesting problems exactly and so you've got that tension of developers wanting to do that uh and and then the other side of the the thing the other thing that's feeding into the tension is just like as andrew was saying like well eventually we're going to need to make money and and like this this like going back and

forth between those two things i mean it's as we were kind of chronically and uh and uh alluding to it's eaten up a lot of little startups and efforts here but it's also resulted in several like great ones uh and and ones that in good innovations in this area so while it's kind of mysterious how that stuff works out i think it's at least in my books it's pretty clear that open source is a good driver to make innovation happen in the industry both in the business model side and and more in the technical side but uh yeah it is uh that that that figuring out as andrew was saying like how you how you harness the desire to solve big problems in the open is is always a bit of a uh some magic how do you how do you avoid the tragedy of the commons is is really the question exactly exactly well just uh just one more thing before we wrap up so i mean you alerted to some stuff that's uh that's coming in the future like uh more attention to um uh alerting and setting up uh things for alerts but when when you kind of look over the next i don't know i don't know if you live yourself a uh live your life a quarter a year at a time or whatever but like when you kind of look out six 12 months or so what are some areas in in uppercase metrics that uh that you think would be interesting to uh work on and get out there yeah so i think i try to look like one one release out at a minimum uh but i i think the once we get get past the alerting stuff um and and i think there are some things that are sort of obvious to most anyone who's looking at a at a metrics and monitoring setup like there are some things that are just sort of expected and we're trying to fill kind of fill in those gaps and i think like i said alerting is the first one um there's going to be a little bit more i think to to allow the platform itself to be slightly more extensible so right now we're kind of limiting um some of the amount of stuff that people can put in primarily because you know we're also deploying uh like i said my sql is kind of the storage back end for this we've got some processing stuff in front of that but basically there is a point where it wouldn't be able to keep up with a certain amount of metrics and you know we're pretty comfortable with where it's at now but it would be nice

for uh people to be able to open it up a little bit more broadly uh control resolution you know if you wanted to be able to store less than one minute roll-ups um things like that and those are kind of just like nice to haves but i think the the bigger thing that i have on my mind and i i haven't really this isn't necessarily uh shared too broadly yet but it's kind of i see that there are a lot of places within the cloud foundry ecosystem where we're kind of leaving a lot of really interesting data uh to to just sort of it's just going nowhere right now so i think there are a lot of lower level things where i think we can pull um some container networking metrics there are lots of things where i think we've got a lot of data within the system that we can start to take advantage of and we can start to build really specialized tools and visualizations to kind of look at you know if one of your let's say one of your nodes gets compromised and somebody's sending a ton of data out over the network you might see a spike on the the outbound networking port and things like that that might be really easy to look at if you're looking at a like a network topology graph but maybe not so easy to look at if you're looking at you know aggregate network traffic so i think there are lots of things like that where we can start to build tools that make you know cloud foundry specifically much more powerful and make the metrics that we're already uh emitting way more valuable to the end users and that's sort of where i feel like the sweet spot of uh it's a combination of our metrics team and the platform metrics team i think once we can start to harness a lot of the data that we already have there will be a ton of value for the customers and the operators in just kind of using cloud foundry better and i think that's that's kind of where i feel like the next six to 12 months are going to be the most interesting is when we actually start to to make the entire system better for the end user yeah it's it's it's uh put it putting uh putting more smi in as it were yeah like fine putting more smi into the 80 of commonality and and exposing all of that exactly i thought it'd be really cool to

be able to expose some of the wash level platform infrastructure metrics to be able to compare the the behavior of infrastructure services i think that we have enough meaningful interaction with a lot of the infrastructure services that we could uh we could do some some really interesting really interesting analysis of how those perform relative to each other yeah and i think i i know the um like i said the plot metrics team i know they're looking at a lot of a lot of those kinds of things that we can we can start to extract that will make it very easy as an operator to just sort of look at a screen i mean initially look at a screen and just say you know red or green is this thing healthy is this thing you know going down and then you know the next step kind of like what we're talking about adding for application metrics they'll i'm sure add some sort of um automated alerting that if you see something within the platform go down you don't have to to wait to check it you get alerted um either as it's happening or as soon as it happens well thanks for being on that was good stuff no problem thank you for having me it's always it's always fun to go down a metrics and monitoring memory lane with myself like i was just as as you're going over extending the uh the raw data if you will over time i was i was remembering i after working in this space for like five years way back when i sort of me and my friends established this principle that like around release three of your systems management project you basically spend all your time managing the database and and figure figuring out how to how to get all that data in there and i mean it goes all the way back to the beginning about how how advances in managing time series data like opens up like uh it frees up a lot of time because uh it's always going to come managing that in a database is difficult it's a problem with the ingest problem with the queries or problem with the storage that's right that's right who knew the problem with computers would be computers uh well uh so i just have a few things i want to call out here at the end uh so we have uh we have got a lot of fun conferences going on uh one of them in particular we just announced uh this coming

out i'm going to be working at this and someone else on andrew and i's uh team mark heckler will be working on it so we had a request in the uh the washington dc area to do a slightly uh government uh oriented um uh one of our little workshops so we're going to be doing that on uh june 7th i'll put a link in the show notes to it but if you're interested in it's it's i think it's like two thirds of a day and uh it's free of course you can come and just hear about uh a lot of the the sort of pivotal way of doing things uh and also we have uh in chicago we have spring days coming up if you go to springdays.io you can see that and that's may 30th and uh not as my notes say 21st it's probably may 30th and 31st uh and there'll also be one of those in new york and atlanta uh that you can you can look up there but that's uh two days of as you would imagine if you're into microservices or development or java or spring or uh or you know just looking not to have to go to your real job it's uh it's a good way to spend those two days of time in chicago new york and atlanta and then finally there's the cf summit the cloud foundry summit uh which goes over how large organizations are doing all their cloud native digital transformation and also the technical side of uh of doing things in a cloud foundry way all the way down to the bottom of the application stop stack up to the top uh more spring oriented and application oriented so that's june 13th to 15th and i'll put this in the show notes as well or you can get out your pencil and write this down but if you want to get 20 off of that you can use the code cfsv17 Coté Coté that's my name uh and then you know later in the year the SpringOne platform but that's that's way far ahead planning uh so with that thanks for listening as always this has been Pivotal Conversations if you want to go look at the archives and and find uh a way to subscribe find the rss feed and all kinds of other doodaddles uh you can go to soundcloud.com slash Pivotal Conversations with no space we also post the full show notes on our blog at pod not podcast at pivotal.io slash podcast and uh i'm Coté in twitter

if you want to follow up with me that's c-o-t-e how about yourself guest where can people scurry around with you on the internet uh i i don't tweet much but um i guess i i do have a twitter account it's just at todd pearson t-o-d-d-p-e-r-s-e-n i'll i'll tweet something eventually that's right and yourself andrew i'm little idea on twitter fastest way to get my attention so uh with that we'll see everyone next time bye bye

Tanzu Talk · 🍕 Get more pizza.